Moving picture coding method, and moving picture decoding method

ABSTRACT

A moving picture coding apparatus for performing inter-picture predictive coding for pictures constituting a moving picture is provided with a coding unit for performing predictive error coding for image data; a decoding unit for performing predictive error decoding for an output from the coding unit; a reference picture memory for holding output data from the decoding unit; and a motion vector detection unit for detecting motion vectors on the basis of the decoded image data stored in the memory. When coding a B picture as a target picture, information indicating whether or not the target picture should be used as a reference picture when coding another picture is added as header information. Therefore, in a decoding apparatus for decoding a bit stream Bs outputted from the moving picture coding apparatus, management of a memory for holding the reference picture can be facilitated on the basis of the header information.

TECHNICAL FIELD

The present invention relates to a moving picture coding method and amoving picture decoding method and, more particularly, to a method forcoding or decoding pictures constituting a moving picture with referenceto other pictures of the moving picture.

BACKGROUND ART

Generally, in coding of pictures constituting a moving picture, eachpicture is divided into plural blocks, and compressive coding(hereinafter, also referred to simply as “coding”) of image informationpossessed by each picture is carried out for every block, utilizingredundancies in the space direction and time direction of the movingpicture. As a coding process utilizing redundancy in the spacedirection, there is intra-picture coding utilizing correlation of pixelvalues in a picture. As a coding process utilizing redundancy in thetime direction, there is inter-picture predictive coding utilizingcorrelation of pixel values between pictures. The inter-picturepredictive coding is a process of coding a target picture to be coded,with reference to a picture that is positioned timewise forward thetarget picture (forward picture), or a picture that is positionedtimewise backward the target picture (backward picture).

The forward picture is a picture whose display time is earlier that thatof the target picture, and it is positioned forward the target pictureon a time axis indicating the display times of the respective pictures(hereinafter, referred to as “display time axis”). The backward pictureis a picture whose display time is later than that of the targetpicture, and it is positioned backward the target picture on the displaytime axis. Further, in the following description, a picture to bereferred to in coding the target picture is called a reference picture.

In the inter-picture predictive coding, specifically, a motion vector ofthe target picture with respect to the reference picture is detected,and prediction data for image data of the target picture is obtained bymotion compensation based on the motion vector. Then, redundancy ofdifference data between the prediction data and the image data of thetarget picture in the space direction of the picture is removed, therebyto perform compressive coding for the amount of data of the targetpicture.

On the other hand, as a process for decoding a coded picture, there areintra-picture decoding corresponding to the intra-picture coding, andinter-picture decoding corresponding to the inter-picture coding. In theinter-picture decoding, the same picture as a picture that is referredto in the inter-picture coding is referred to. That is, a picture Xtgthat is coded with reference to pictures Xra and Xrb is decoded withreference to the pictures Xra and Xrb.

FIGS. 43(a)-43(c) are diagrams illustrating plural pictures constitutinga moving picture.

In FIG. 43(a), part of plural pictures constituting one moving pictureMpt, i.e., pictures F(k)˜F(k+2n−1) [k,n: integers], are shown. Displaytimes t(k)˜t(k+2n−1) are set on the respective pictures F(k)˜F(k+2n−1).As shown in FIG. 43(a), the respective pictures are successivelyarranged from one having earlier display time on a display time axis Xindicating display times Tdis of the respective pictures, and thesepictures are grouped for every predetermined number (n) of pictures.Each of these picture groups is called a GOP (Group of Pictures), andthis is a minimum unit of random access to coded data of a movingpicture. In the following description, a picture group is sometimesabbreviated as a GOP.

For example, an (i)th picture group Gp(i) is constituted by picturesF(k)˜F(k+n−1). An (i+1)th picture group Gp(i+1) is constituted bypictures F(n+k)˜F(k+2n−1).

Each picture is divided into plural slices each comprising pluralmacroblocks. For example, a macroblock is a rectangle area having 16pixels in the vertical direction and 16 pixels in the horizontaldirection. Further, as shown in FIG. 43(b), a picture F(k+1) is dividedinto plural slices SL1˜SLm [m: natural number]. A slice SL2 isconstituted by plural macroblocks MB1˜MBr [r: natural number] as shownin FIG. 43(c).

FIG. 44 is a diagram for explaining coded data of a moving picture,illustrating a structure of a stream obtained by coding the respectivepictures constituting the moving picture.

A stream Smp is coded data corresponding to one image sequence (e.g.,one moving picture). The stream Smp is composed of an area (commoninformation area) Cstr wherein bit streams corresponding to commoninformation such as a header are arranged, and an area (GOP area) Dgopwherein bit streams corresponding to the respective GOPs are arranged.The common information area Cstr includes sync data Sstr and a headerHstr corresponding to the stream. The GOP area Dgop includes bit streamsBg(1)˜Bg(i−1), Bg(i), Bg(i+1)˜Bg(I) corresponding to picture groups(GOP) Gp(1)˜Gp(i−1), Gp(i), Gp(i+1)˜Gp(I) [i,I: integers].

Each bit stream corresponding to each GOP is composed of an area (commoninformation area) Cgop wherein bit streams corresponding to commoninformation such as a header are arranged, and an area (picture area)Dpct wherein bit streams corresponding to the respective pictures arearranged. The common information area Cgop includes sync data Sgop and aheader Hgop corresponding to the GOP. A picture area Dpct of the bitstream Bg(i) corresponding to the picture group G(i) includes bitstreams Bf(k′), Bf(k′+1), Bf(k′+2), Bf(k′+3), . . . , Bf(k′+s)corresponding to pictures F(k′), F(k′+1), F(k′+2), F(k′+3), . . . ,F(k′+s) [k′,s: integers]. The pictures F(k′), F(k′+1), F(k′+2), F(k′+3),. . . , F(k′+s) are obtained by rearranging, in coding order, thepictures F(k)˜F(k+n−1) arranged in order of display times.

Each bit stream corresponding to each picture is composed of an area(common information area) Cpct wherein bit streams corresponding tocommon information such as a header are arranged, and an area (slicearea) Dslc wherein bit streams corresponding to the respective slicesare arranged. The common information area Cpct includes sync data Spctand a header Hpct corresponding to the picture. For example, when thepicture F(k′+1) in the arrangement in order of coding times (codingorder arrangement) is the picture F(k+1) in the arrangement in order ofdisplay times (display order arrangement), the slice area Dslc in thebit stream Bf(k′+1) corresponding to the picture F(k′+1) includes bitstreams Bs1˜Bsm corresponding to the respective slices SL1˜SLm.

Each bit stream corresponding to each slice is composed of an area(common information area) Cs1 c wherein bit streams corresponding tocommon information such as a header are arranged, and an area(macroblock area) Dmb wherein bit streams corresponding to therespective macroblocks are arranged. The common information area Cs1 cincludes sync data Sslc and a header Hslc corresponding to the slice.For example, when the picture F(k′+1) in the coding order arrangement isthe picture F(k+1) in the display order arrangement, the macroblock areaDmb in the bit stream Bs2 corresponding to the slice SL2 includes bitstreams Bm1˜Bmr corresponding to the respective macroblocks MB1˜MBr.

As described above, coded data corresponding to one moving picture(i.e., one image sequence) has a hierarchical structure comprising astream layer corresponding to a stream Smp as the coded data, GOP layerscorresponding to GOPs constituting the stream, picture layerscorresponding to pictures constituting each of the GOPs, and slicelayers corresponding to slices constituting each of the pictures.

By the way, in moving picture coding methods such as MPEG (MovingPicture Experts Group)-1, MPEG-2, MPEG-4, ITU-T recommendation H.263,H.26L, and the like, a picture to be subjected to intra-picture codingis called an I picture, and a picture to be subjected to inter-picturepredictive coding is called a P picture or a B picture.

Hereinafter, definitions of an I picture, a P picture, and a B picturewill be described.

An I picture is a picture to be coded without referring to anotherpicture. A P picture or B picture is a picture to be coded withreference to another picture. To be exact, a P picture is a picture forwhich either I mode coding or P mode coding can be selected when codingeach block in the picture. A B picture is a picture for which one of Imode coding, P mode coding, and B mode coding can be selected whencoding each block in the picture.

The I mode coding is a process of performing intra-picture coding for atarget block in a target picture without referring to another picture.The P mode coding is a process of performing inter-picture predictivecoding for a target block in a target picture with reference to analready-coded picture. The B mode coding is a process of performinginter-picture predictive coding for a target block in a target picturewith reference to two already-coded pictures.

A picture to be referred to during the P mode coding or B mode coding isan I picture or a P picture other than the target picture, and it may beeither a forward picture positioned forward the target picture or abackward picture positioned backward the target picture.

However, there are three ways of combining two pictures to be referredto during the B mode coding. That is, there are three cases of B modecoding as follows: a case where two forward pictures are referred to, acase where two backward pictures are referred to, and a case where oneforward picture and one backward picture are referred to.

FIG. 45 is a diagram for explaining a moving picture coding method suchas MPEG described above. FIG. 45 illustrates relationships betweentarget pictures and the corresponding reference pictures (pictures to bereferred to when coding the respective target pictures).

Coding of the respective pictures F(k)˜F(k+7), . . . , F(k+17)˜F(k+21)constituting the moving picture is carried out with reference to otherpictures as shown by arrows Z. To be specific, a picture at the end ofone arrow Z is coded by inter-picture predictive coding with referenceto a picture at the beginning of the same arrow Z. In FIG. 45, thepictures F(k)˜F(k+7), . . . , F(k+17)˜F(k+21) are identical to thepictures F(k)˜F(k+4), . . . , F(k+n−2)˜F(k+n+4), . . . , F(k+2n−2),F(k+2n−1) shown in FIG. 43(a). These pictures are successively arrangedfrom one having earlier display time on the display time axis X. Thedisplay times of the pictures F(k)˜F(k+7), . . . , F(k+17)˜F(k+21) aretimes t(k)˜t(k+7), . . . , t(k+17)˜t(k+21). The picture types of thepictures F(k)˜F(k+7) are I, B, B, P, B, B, P, B, and the picture typesof the pictures F(k+17)˜F(k+21) are B, P, B, B, P.

For example, when performing B mode coding for the second B pictureF(k+1) shown in FIG. 45, the first I picture F(k) and the fourth Ppicture F(k+3) are referred to. Further, when performing P mode codingfor the fourth P picture F(k+3) shown in FIG. 45, the first I pictureF(k) is referred to.

Although a forward picture is referred to in P mode coding of a Ppicture in FIG. 45, a backward picture may be referred to. Further,although a forward picture and a backward picture are referred to in Bmode coding of a B picture in FIG. 45, two forward pictures or twobackward pictures may be referred to.

Furthermore, in a moving picture coding method such as MPEG-4 or H.26L,a coding mode called “direct mode” may be selected when coding a Bpicture.

FIGS. 46(a) and 46(b) are diagrams for explaining inter-picturepredictive coding to be performed with the direct mode. FIG. 46(a) showsmotion vectors to be used in the direct mode.

In FIG. 46(a), pictures P1, B2, B3, and P4 correspond to the picturesF(k+3)˜F(k+6) [k=−2] shown in FIG. 45, and times t(1), t(2), t(3), andt(4) (t(1)<t(2)<t(3)<t(4)) are display times of the pictures P1, B2, B3,and P4, respectively. Further, X is a display time axis indicatingdisplay times Tdis.

Hereinafter, a case where a block BL3 in the picture B3 is coded in thedirect mode will be specifically described.

In this case, a target picture to be coded is the picture B3, and atarget block to be coded is a block BL3.

In predictive coding of the block BL3 in the picture B3, a motion vectorMV4 of a block BL4 in the picture P4, which block has been most-recentlycoded and is positioned backward the picture B3, is used. The relativeposition of the block BL4 to the picture P4 is equal to the relativeposition of the block BL3 to the picture B3. That is, as shown in FIG.46(b), coordinates (x4,y4) of an origin Ob4 of the block BL4 withrespect to an origin O4 of the picture P4 are equal to coordinates(x3,y3) of an origin Ob3 of the block BL3 with respect to an origin O3of the picture P3. Further, the motion vector MV4 of the block BL4 isthe motion vector that is used in predictive coding of the block BL4.The motion vector MV4 of the block BL4 is obtained by motion detectionof the block BL4 with reference to the forward picture P1, and it showsa region R4 f corresponding to the block BL4, of the forward picture P1.

Then, the block BL3 in the picture B3 is subjected to bidirectionalpredictive coding with reference to the forward picture P1 and thebackward picture P4, by using motion vectors MV3 f and MV3 b which areparallel to the motion vector MV4. The motion vector MV3 f indicates aregion R3 f corresponding to the block BL3, of the forward picture P1 tobe referred to when coding the block BL3. The motion vector MV3 bindicates a region R3 b corresponding to the block BL3, of the backwardpicture P4 to be referred to when coding the block BL3.

By the way, the ITU-T recommendation (H.263++ Annex U) describes about aframework in a case where plural pictures are used as candidates for areference picture. In this description, a reference picture memory forholding image data of pictures to be candidates for a reference picture(candidate pictures) is sorted into a short-term picture memory and along-term picture memory. The short-term picture memory is a memory areafor holding data of candidate pictures which are timewise close to atarget picture (neighboring candidate pictures). The long-term picturememory is a memory area for holding candidate pictures which aretimewise far from the target picture (distant candidate pictures). To bespecific, a distant candidate picture is apart from the target pictureby such a distance that the number of candidate pictures from the targetpicture to the distant candidate picture exceeds the number of candidatepictures which can be stored in the short-term picture memory.

Further, the ITU-T recommendation (H.263++ Annex U) describes about amethod of utilizing the short-term picture memory and the long-termpicture memory, and further, it also describes a method of designatingreference picture indices (hereinafter, also referred to simply asreference indices) to pictures.

Initially, the method of designating reference indices to pictures willbe briefly described.

FIGS. 47(a) and 47(b) are diagrams for explaining the method ofdesignating reference indices to plural pictures constituting a movingpicture. FIG. 47(a) shows candidates (candidate pictures) for a pictureto be referred to when coding a picture P16. FIG. 47(b) shows candidates(candidate pictures) for a picture to be referred to when coding apicture B15.

In FIG. 47(a), pictures P4, B2, B3, P7, B5, B6, P10, B8, B9, P13, B11,B12, P16, B14, B15, P19, B17, and P18 are obtained by rearranging thepictures F(k+1)˜F(k+17) [k=1] shown in FIG. 45 in cording order. Thearrangement of plural pictures shown in FIG. 47(a) is an arrangement ofpictures on a time axis (coding time axis) Y indicating times (codingtimes) Tenc for coding the respective pictures.

A description will be given of a case where, as shown in FIG. 47(a), ablock in the P picture P16 is subjected to P mode coding.

In this case, among four forward P pictures (pictures P4, P7, P10, andP13), a picture suited for coding is referred to. That is, the forward Ppictures P4, P7, P10, and P13 are candidate pictures which can bedesignated as a reference picture in performing P mode coding of thepicture P16. These candidate pictures P4, p7, P10, and P13 are assignedreference indices, respectively.

When assigning reference indices to these candidate pictures, areference index having a smaller value is assigned to a candidatepicture closer to the target picture P16 to be coded. To be specific, asshown in FIG. 47(a), reference indices [0], [1], [2], and [3] areassigned to the pictures P13, P10, P7, and P4, respectively. Further,information indicating the reference indices assigned to the respectivecandidate pictures is described as a parameter of motion compensation ina bit stream corresponding to a target block in the picture p16.

Next, a description will be given of a case where, as shown in FIG.47(b), a block in the B picture B15 is subjected to B mode coding.

In this case, among four forward pictures (pictures P4, P7, P10, andP13) and one backward picture (picture P16), two pictures suited forcoding are referred to. That is, the forward pictures P4, P7, P10, andP13 and the backward picture P16 are candidate pictures which can bedesignated as reference pictures in B mode coding for the B picture B15.When four forward pictures and one backward picture are candidatepictures, the forward pictures P4, P7, P10, and P13 are assignedreference indices, and the backward picture P16 is assigned a code [b]indicating that this picture is a candidate picture to be referred tobackward.

In assigning reference indices to the candidate pictures, as for forwardpictures as candidate pictures, a smaller reference index is assigned toa forward picture (candidate picture) closer to the target picture B15to be coded on the coding time axis Y. To be specific, as shown in FIG.47(b), reference indices [0], [1], [2], and [3] are assigned to thepictures P13, P10, P7, and P4, respectively. Further, informationindicating the reference index assigned to each candidate picture isdescribed, as a parameter of motion picture, in a bit streamcorresponding to a target block in the picture B15.

Next, the method of assigning reference indices, which is described inthe ITU-T recommendation (H.263++ Annex U), will be described inassociation with the method of utilizing the short-term picture memoryand the long-term picture memory.

In the short-term picture memory, candidate pictures which can bedesignated as a reference picture for a target picture are successivelystored, and the stored candidate pictures are assigned reference indexin order of storage into the memory (i.e., in decoding order, or inorder of bit streams). Further, when decoding a B picture, a picturethat has most-recently been stored in the memory is treated as abackward reference picture while the other pictures are assignedreference indices in order of storage into the memory.

Hereinafter, a description will be given of a case where four forwardpictures can be used as candidates for a reference picture for a targetpicture.

FIGS. 48(a) and 48(b) are diagrams illustrating part of plural picturesconstituting a moving picture, wherein pictures are arranged in displayorder (48(a)), and pictures are arranged in coding order (48(b)).Pictures P1, B2, B3, P4, B5, B6, P7, B8, B9, P10, B11, B12, P13, B14,B15, P16, B17, B18, and P19 shown in FIG. 48(a) correspond to thepictures F(k+3)˜F(k+21) [k=−2] shown in FIG. 45.

FIG. 49 is a diagram for explaining management of a memory for referencepictures for the pictures arranged as described above.

In FIG. 49, already-coded pictures which are stored in the referencepicture memory when coding target pictures are shown in association withlogical memory numbers corresponding to memory areas where thealready-coded pictures are stored, and reference indices assigned to thealready-coded pictures.

In FIG. 49, pictures P16, B14, and B15 are target pictures. Logicalmemory numbers (0)˜(4) indicate logical positions (memory areas) in thereference picture memory. The later the time of coding (or decoding) analready-processed picture stored in a memory area is, the smaller thelogical memory number corresponding to the memory area is.

Hereinafter, management of the reference picture memory will bedescribed more specifically.

When coding (decoding) the picture P16, the pictures P13, P10, P7, andP4 are stored in the memory areas indicated by the logical memorynumbers (0), (1), (2), and (3) in the reference picture memory,respectively. The pictures P13, P10, P7, and P4 are assigned referenceindices [0], [1], [2], and [3], respectively.

When coding (decoding) the pictures B14 and B15, the pictures P16, P13,P10, P7, and P4 are stored in the memory areas indicated by the logicalmemory numbers (0), (1), (2), (3), and (4) in the reference picturememory, respectively. At this time, the picture P16 is assigned a code[b] indicating that this picture is a candidate picture to be backwardreferred to, and the remaining candidate pictures p13, P10, P7, and P4to be forward referred to are assigned reference indices [0], [1], [2],and [3], respectively.

Information indicating the reference indices assigned to the respectivecandidate pictures is a parameter of motion compensation and, whencoding a block in a target picture, it is described in a bit streamcorresponding to the block as information indicating which one of theplural candidate pictures should be used as a reference picture. At thistime, a shorter code is assigned to a smaller reference index.

In the conventional coding method described above, however, since an Ipicture or a P picture is designated as a reference picture whenperforming predictive coding for a block in a B picture, a distance(hereinafter, also referred to as a time-basis distance) between thetarget picture and the reference picture on the display time axis mightbe increased.

For example, in predictive coding on a block in the B picture B15 shownin FIG. 48(b), when the forward picture P13 and the backward picture P16are designated as reference pictures, the time-basis distance Ltd(=t(15)−t(13)) between the B picture B15 (target picture) and theforward picture P13 (reference picture) becomes a two-picture interval(2Pitv) as shown in FIG. 50(a).

Furthermore, in predictive coding for a block in the B picture B15 shownin FIG. 48(b), when the forward pictures P13 and P10 are designated asreference pictures, the time-basis distance Ltd (=t(15)-t(10)) betweenthe B picture B15 (target picture) and the forward picture P10(reference picture) becomes a five-picture interval (5Pitv) as shown inFIG. 50(b).

Especially when the number of B pictures inserted between an I pictureand a P picture or between adjacent two P pictures is increased, thetime-basis distance Ltd between the target picture and the referencepicture is increased, resulting in a considerable reduction in codingefficiency.

Further, in the conventional coding method, when performing B modecoding in which plural backward pictures can be referred to, there arecases where a neighboring picture which is timewise close to a targetpicture is assigned a reference index larger than a reference indexassigned to a distant picture which is timewise far from the targetpicture.

In this case, in motion detection for a block in the target picture, acandidate picture that is timewise closer to the target picture islikely to be referred to, in other words, a candidate picture that istimewise closer to the target picture is likely to be designated as areference picture, resulting in degradation of coding efficiency.

Hereinafter, a description will be given of a case where two backwardpictures P16 and p19 are referred to in B mode coding for a block in a Bpicture B15 shown in FIG. 51(a).

In this case, pictures B2, B3, P4, B5, B6, P7, B8, B9, P10, B11, B12,P13, B14, B15, P16, B17, B18, and P19 which are arranged in displayorder as shown in FIG. 51(a) are rearranged in coding order, resultingin P7, B2, B3, P10, B5, B6, P13, B8, B9, P16, B11, B12, P19, B14, andB15 as shown in FIG. 51(b).

Further, in this case, among three forward pictures (pictures P7, P10,and P13) and two backward pictures (pictures P16 and P19), two picturessuited to coding are referred to. To be specific, the forward picturesp7, P10, and P13 and the backward pictures P16 and P19 are candidatepictures which can be designated as a reference picture when coding ablock in the picture B15. When three forward pictures and two backwardpictures are candidate pictures as described above, reference indicesare assigned to the forward pictures P7, P10, and P13 and the backwardpictures P16 and P19.

In assigning reference indices to the candidate pictures, a smallerreference index is assigned to a candidate picture that is closer to thetarget picture B15 to be coded on the coding time axis Y. To bespecific, as shown in FIG. 51(b), reference indices [0], [1], [2], [3],and [4] are assigned to the pictures P19, P16, P13, P10, and P7,respectively.

In this case, however, the reference index [1] assigned to the P pictureP16 that is closer to the target picture (B picture B15) on the displaytime axis X becomes larger than the reference index [0] assigned to theP picture P19 that is far from the B picture B15, resulting indegradation of coding efficiency.

The present invention is made to solve the above-described problems andhas for its object to provide a moving picture coding method which canprevent a reduction in coding efficiency due to an increase in atime-basis distance between a target picture and a reference picture,and a moving picture decoding method corresponding to the moving picturecoding method which can prevent a reduction in coding efficiency.

Further, it is another object of the present invention to provide amoving picture coding method which can assign reference indices tocandidate pictures that can be referred to in predictive coding, withoutdegrading coding efficiency, and a moving picture decoding methodcorresponding to the moving picture coding method which can avoiddegradation in coding efficiency.

DISCLOSURE OF THE INVENTION

According to the present invention, there is provided a moving picturecoding method for dividing each of plural pictures constituting a movingpicture into plural blocks, and coding each picture for every block,which method includes a coding step of performing predictive coding fora block of a target picture to be coded, with reference to analready-coded picture; and, in the coding step, when the target pictureis a B picture whose block is to be predictive-coded with reference totwo already-coded pictures, a block of the target picture ispredictive-coded with reference to an already-coded B picture.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when the target picture is aB picture, a block of the target picture is predictive-coded withreference to an already-coded B picture, and when the target picture isa P picture whose block is to be predictive-coded with reference to onealready-coded picture, each block of the target picture ispredictive-coded without referring to any already-coded B picture.

According to the present invention, in the above-described movingpicture coding method, each of the plural pictures constituting themoving picture is, in the coding step, coded as one of the followingpictures: an I picture whose block is to be coded without referring toan already-coded picture, a P picture whose block is to bepredictive-coded with reference to one already-coded picture, and a Bpicture whose block is to be predictive-coded with reference to twoalready-coded pictures; and, in the coding step, when the target pictureis a B picture, a block of the target picture is predictive-coded withreference to an already-coded B picture, and when the number ofcandidate pictures for a reference picture to be referred to when codingthe target picture as a B picture is equal to or smaller than the numberof candidate pictures for a reference picture to be referred to whencoding the target picture as a P picture.

According to the present invention, in the above-described movingpicture coding method, each of the plural pictures constituting themoving picture is, in the coding step, coded as one of the followingpictures: an I picture whose block is to be coded without referring toan already-coded picture, a P picture whose block is to bepredictive-coded with reference to one already-coded picture, and a Bpicture whose block is to be predictive-coded with reference to twoalready-coded pictures; and, in the coding step, when the target pictureis a B picture, a B picture to be referred to in predictive-coding ablock of the target picture is only a B picture which is insertedbetween the target picture and an I or a P picture that is closest tothe target picture on the display time axis.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when the target picture is aB picture, a block of the target picture is predictive-coded withreference to an already-coded B picture, and picture positioninformation indicating the position of the referred already-coded Bpicture on the display time axis, is included in a bit stream that isobtained by coding the pictures constituting the moving picture.

According to the present invention, in the above-described movingpicture coding method, the picture position information is expressedwith a shorter length code as the distance on the display time axis fromthe target picture to the already-coded B picture that is referred to incoding the target picture is shorter.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when the target picture is aB picture, a block of the target picture is predictive-coded withreference to an already-coded B picture; and information indicating thatthe already-coded B picture is referred to when coding the target Bpicture, is included as header information in a bit stream that isobtained by coding the pictures constituting the moving picture.

According to the present invention, there is provided a moving picturecoding method for dividing each of plural pictures constituting a movingpicture into plural blocks, and coding each picture for every block,which method includes a coding step of coding a target picture to becoded, with reference to, at least, a P picture whose block is to bepredictive-coded with reference to one already-coded picture, and a Bpicture whose block is to be predictive-coded with reference to twoalready-coded pictures; and, in the coding step, an already-codedpicture determined according to a certain rule is referred to whencoding a target block of a B picture as a target picture in a directmode which uses a motion vector of a base block that is located atspatially the same position as the target block, in an already-codedbase picture that is positioned close to the target picture.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when coding the target blockin the direct mode, a first already-coded picture which is positionedimmediately before the target picture and is earlier in display orderthan the target picture, is referred to.

According to the present invention, in the above-described movingpicture coding method, the already-coded base picture including the baseblock is a backward base P picture which is later in display order thanthe target picture; and, in the coding step, a forward motion vector(MVR×TRF/TRD) and a backward motion vector ((TRB−TRD)×MVR/TRD) of thetarget block are obtained, on the basis of a magnitude MVR of the motionvector of the base block, a distance TRD between the backward base Ppicture and a second picture which is referred to in coding the baseblock, on the display time axis, a distance TRF between the targetpicture and the first picture on the display time axis, and a distanceTRB between the target picture and the second picture on the displaytime axis, and bidirectional prediction is carried out using the forwardmotion vector and the backward motion vector.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when coding the target blockin the direct mode, bidirectional prediction with the motion vector ofthe target block being zero is carried out, with reference to analready-coded forward picture which is positioned closest to the targetpicture and is earlier in display order than the target picture, and analready-coded backward picture which is positioned closest to the targetpicture and is later in display order than the target picture.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when coding the target blockin the direct mode, no image information of the target block whoseprediction error information becomes zero, into the bit streamcorresponding to the moving picture, is inserted.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when the prediction errorinformation of the target block becomes zero, insertion of the imageinformation of the target block into the bit stream corresponding to themoving picture, is omitted.

According to the present invention, in the above-described movingpicture coding method, in the coding step, reference picture indices areassigned to candidate pictures for a reference picture to be referred towhen coding the target picture, and when coding the target block in thedirect mode, a candidate picture to which a specific reference pictureindex is assigned is referred to.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when a picture immediatelybefore the target picture is a picture to be used as a candidate picturefor a reference picture only when coding the target picture, thespecific reference picture index is assigned to a picture which ispositioned forward the target picture, except the picture immediatelybefore the target picture, among the candidate pictures to be referredto in coding the target picture.

According to the present invention, in the above-described movingpicture coding method, in the coding step, the specific referencepicture index is assigned to a candidate picture which is earlier indisplay order than the target picture and is closest to the targetpicture, except the picture immediately before the target picture, amongthe candidate pictures to be referred to in decoding the target picture.

According to the present invention, in the above-described movingpicture coding method, the already-coded base picture including the baseblock is a backward base P picture which is later in display order thanthe target picture; and, in the coding step, when coding the targetblock in the direct mode, a first forward picture which is earlier indisplay order than the target block, which is referred to in coding thebase block, is referred to.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when coding the target blockin the direct mode, a second forward picture which is positionedimmediately before the target picture and is earlier in display orderthan the target picture, is referred to; and a forward motion vector(MVR×TRF/TRD) and a backward motion vector ((TRB−TRD)×MVR/TRD) of thetarget block are obtained, on the basis of a magnitude MVR of the motionvector of the base block, a distance TRD between the backward base Ppicture and the first forward picture on the display time axis, adistance TRF between the target picture and the second forward pictureon the display time axis, and a distance TRB between the target pictureand the first forward picture on the display time axis.

According to the present invention, in the above-described movingpicture coding method, in the coding step, when coding the target blockin the direct mode, if a forward picture to be referred to, which isearlier in display order than the target picture, does not exist in amemory for holding reference pictures, a picture which is closest to thetarget picture and is earlier in display order than the target picture,is referred to.

According to the present invention, there is provided a moving picturecoding method for coding each of plural pictures constituting a movingpicture to generate a bit stream corresponding to each picture, whichmethod includes a coding step of coding a target picture to be coded,with reference to an already-coded picture; and the coding stepincludes: an index assignment step of assigning reference pictureindices to plural reference candidate pictures which are candidates fora reference picture to be referred to in coding the target picture, insuch a manner that a smaller reference picture index is assigned to acandidate picture which is closer in display order to the target pictureto be coded, and an index addition step of adding the reference pictureindex which is assigned to a picture that is referred to in coding thetarget picture, to the bit stream.

According to the present invention, there is provided a moving picturecoding method for coding each of plural pictures constituting a movingpicture to generate a bit stream corresponding to each picture, whichmethod includes a coding step of coding a target picture to be codedwith reference to an already-coded picture and, in the coding step, aflag indicating whether or not the target picture should be used as acandidate for a reference picture when coding another picture thatfollows the target picture, is described in the bit stream.

According to the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture, for every block that is a processing unit of eachpicture, thereby converting a bit stream corresponding to each pictureinto image data, which method includes a decoding step of performingpredictive decoding for a block of a target picture to be decoded, withreference to an already-decoded picture; and, in the decoding step, whenthe target picture is a B picture whose block is to bepredictive-decoded with reference to two already-decoded pictures, ablock of the target picture is predictive-decoded with reference to analready-decoded B picture.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when the target pictureis a B picture, a block of the target picture is predictive-decoded withreference to an already-decoded B picture, and when the target pictureis a P picture whose block is to be predictive-decoded with reference toone already-decoded picture, each block of the target picture ispredictive-decoded without referring to any already-decoded B picture.

According to the present invention, in the above-described movingpicture decoding method, each of the plural pictures constituting themoving picture is, in the decoding step, decoded as one of the followingpictures: an I picture whose block is to be decoded without referring toan already-decoded picture, a P picture whose block is to bepredictive-decoded with reference to one already-decoded picture, and aB picture whose block is to be predictive-decoded with reference to twoalready-decoded pictures; and, in the decoding step, when the targetpicture is a B picture, a block of the target picture ispredictive-decoded with reference to an already-decoded B picture, andthe number of candidate pictures for a forward reference picture to bereferred to when decoding the target picture as a B picture is equal toor smaller than the number of candidate pictures for a reference pictureto be referred to when decoding target picture as a P picture.

According to the present invention, in the above-described movingpicture decoding method, each of the plural pictures constituting themoving picture is, in the decoding step, decoded as one of the followingpictures: an I picture whose block is to be decoded without referring toan already-decoded picture, a P picture whose block is to bepredictive-decoded with reference to one already-decoded picture, and aB picture whose block is to be predictive-decoded with reference to twoalready-decoded pictures; and, in the decoding step, when the targetpicture is a B picture, a B picture to be referred to inpredictive-decoding a block of the target picture is only a B picturewhich is inserted between the target picture and an I or a P picturethat is closest to the target picture in display order.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when the target pictureis a B picture, a process of predictive-decoding a block of the targetpicture with reference to an already-decoded B picture, is carried outon the basis of picture position information indicating the position ofthe already-decoded B picture on the display time axis, whichinformation is included in the bit stream.

According to the present invention, in the above-described movingpicture decoding method, the picture position information is expressedwith a shorter length code as the distance on the display time axis fromthe target picture to the already-decoded forward B picture that isreferred to in decoding the target picture is shorter.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when the target pictureis a B picture, a process of predictive-decoding a block of the targetpicture with reference to an already-decoded B picture, is carried outwith reference to header information indicating that an already-coded Bpicture is referred to when coding the target B picture, which headerinformation is included in the bit stream corresponding to the pictureas a component of the moving picture.

According to the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture, for every block that is a processing unit of eachpicture, thereby converting a bit stream corresponding to each pictureinto image data, which method includes a decoding step of decoding atarget picture to be decoded, with reference to, at least, a P picturewhose block is to be predictive-decoded with reference to onealready-decoded picture, and a B picture whose block is to bepredictive-decoded with reference to two already-decoded pictures; and,in the decoding step, an already-decoded picture determined according toa certain rule is referred to when decoding a target block of a Bpicture as a target picture in a direct mode which uses a motion vectorof a base block that is located at spatially the same position as thetarget block, in an already-decoded base picture that is positionedclose to the target picture.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when decoding the targetblock in the direct mode, a first already-coded picture which ispositioned immediately before the target picture and is earlier indisplay order than the target picture, is referred to.

According to the present invention, in the above-described movingpicture decoding method, the already-decoded base picture including thebase block is a backward base P picture which is later in display orderthan the target picture; and, in the decoding step, a forward motionvector (MVR×TRF/TRD) and a backward motion vector ((TRB−TRD)×MVR/TRD) ofthe target block are obtained, on the basis of a magnitude MVR of themotion vector of the base block, a distance TRD between the backwardbase P picture and a second picture which is referred to in decoding thebase block, on the display time axis, a distance TRF between the targetpicture and the first picture on the display time axis, and a distanceTRB between the target picture and the second picture on the displaytime axis.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when decoding the targetblock in the direct mode, bidirectional prediction with the motionvector of the target block being zero is carried out, with reference toan already-decoded forward picture which is positioned closest to thetarget picture and is earlier in display order than the target picture,and an already-decoded backward picture which is positioned closest tothe target picture and is later in display order than the targetpicture.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when decoding the targetblock in the direct mode, image information of the target block whoseprediction error information is zero, which image information is notincluded in the bit stream, is restored using the motion vector of thebase block.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, image information of thetarget block whose prediction error information is zero, which imageinformation is not included in the bit stream, is restored using themotion vector of the base block.

According to the present invention, in the above-described movingpicture decoding method, reference picture indices are assigned tocandidate pictures for a reference picture to be referred to whendecoding the target picture; and, in the decoding step, when decodingthe target block in the direct mode, a candidate picture to which aspecific reference picture index is assigned is referred to.

According to the present invention, in the above-described movingpicture decoding method, when a picture immediately before the targetpicture is a picture to be used as a candidate picture for a referencepicture only when decoding the target picture, the specific referencepicture index is assigned to a target picture which is positionedforward the target picture, except the picture immediately before thetarget picture, among the candidate pictures to be referred to indecoding the target picture; and, in the decoding step, when decodingthe target block in the direct mode, the picture to which the specificreference picture index is assigned is referred to.

According to the present invention, in the above-described movingpicture decoding method, the specific reference picture index isassigned to a candidate picture which is earlier in display order thanthe target picture and is closest to the target picture, except thepicture immediately before the target picture, among the candidatepictures to be referred to in decoding the target picture; and, in thedecoding step, when decoding the target block in the direct mode, thepicture to which the specific reference picture index is assigned isreferred to.

According to the present invention, in the above-described movingpicture decoding method, the already-decoded base picture including thebase block is a backward base P picture which is later in display orderthan the target picture; and, in the decoding step, when decoding thetarget block in the direct mode, a first forward picture which isearlier in display order than the target block, which is referred to indecoding the base block, is referred to.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when decoding the targetblock in the direct mode, a second forward picture which is positionedimmediately before the target picture and is earlier in display orderthan the target picture, is referred to; and a forward motion vector(MV×TRF/TRD) and a backward motion vector ((TRB−TRD)×MVR/TRD) of thetarget block are obtained, on the basis of a magnitude MVR of the motionvector of the base block, a distance TRD between the backward base Ppicture and the first forward picture on the display time axis, adistance TRF between the target picture and the second forward pictureon the display time axis, and a distance TRB between the target pictureand the first forward picture on the display time axis.

According to the present invention, in the above-described movingpicture decoding method, in the decoding step, when decoding the targetblock in the direct mode, if a forward picture to be referred to, whichis earlier in display order than the target picture, does not exist in amemory for holding reference pictures, a picture which is closest to thetarget picture and is earlier in display order than the target pictureis referred to.

According to the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture to convert a bit stream corresponding to each pictureinto image data, which method includes a decoding step of decoding atarget picture to be decoded, with reference to an already-decodedpicture; and the decoding step includes: an index assignment step ofassigning reference picture indices to plural reference candidatepictures which are candidates for a reference picture to be referred toin decoding the target picture, in such a manner that a smallerreference picture index is assigned to a candidate picture which iscloser in display order to the target picture to be decoded, and areference picture determination step of determining a picture to bereferred to in decoding the target picture, on the basis of a referencepicture index assigned to a picture that is referred to in coding thetarget picture, which index is added to the bit stream of the targetpicture, and the reference picture indices assigned to the referencecandidate pictures in the index assignment step.

According to the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture to convert a bit stream corresponding to each pictureinto image data, which method includes a decoding step of decoding atarget picture to be decoded with reference to an already-decodedpicture, wherein a flag indicating whether or not the target pictureshould be used as a candidate for a reference picture when decodinganother picture that follows the target picture, is described in the bitstream and, in the decoding step, management of the decoded targetpicture is carried out on the basis of the flag.

As described above, in the present invention, there is provided a movingpicture coding method for dividing each of plural pictures constitutinga moving picture into plural blocks, and coding each picture for everyblock, which method includes a coding step of performing predictivecoding for a block of a target picture to be coded, with reference to analready-coded picture; and, in the coding step, when the target pictureis a B picture whose block is to be predictive-coded with reference totwo already-coded pictures, a block of the target picture ispredictive-coded with reference to an already-coded B picture.Therefore, when coding a B picture, a forward reference picture that isclosest to the B picture can be used. Thereby, prediction accuracy inmotion compensation for the B picture can be improved, resulting inimproved coding efficiency.

In the above-described moving picture coding method, in the coding step,when the target picture is a B picture, a block of the target picture ispredictive-coded with reference to an already-coded B picture, and whenthe target picture is a P picture whose block is to be predictive-codedwith reference to one already-coded picture, each block of the targetpicture is predictive-coded without referring to any already-coded Bpicture. Therefore, in a memory where pictures to be candidates for areference picture are stored, management of the candidate pictures isfacilitated.

In the above-described moving picture coding method, each of the pluralpictures constituting the moving picture is, in the coding step, codedas one of the following pictures: an I picture whose block is to becoded without referring to an already-coded picture, a P picture whoseblock is to be predictive-coded with reference to one already-codedpicture, and a B picture whose block is to be predictive-coded withreference to two already-coded pictures; and, in the coding step, whenthe target picture is a B picture, a block of the target picture ispredictive-coded with reference to an already-coded B picture, and whenthe number of candidate pictures for a reference picture to be referredto when coding the target picture as a B picture is equal to or smallerthan the number of candidate pictures for a reference picture to bereferred to when coding the target picture as a P picture. Therefore, itis possible to avoid an increase in capacity of the memory holding thereference candidate pictures, which is caused by that another B pictureis referred to when coding a B picture.

In the above-described moving picture coding method, each of the pluralpictures constituting the moving picture is, in the coding step, codedas one of the following pictures: an I picture whose block is to becoded without referring to an already-coded picture, a P picture whoseblock is to be predictive-coded with reference to one already-codedpicture, and a B picture whose block is to be predictive-coded withreference to two already-coded pictures; and, in the coding step, whenthe target picture is a B picture, a B picture to be referred to inpredictive-coding a block of the target picture is only a B picturewhich is inserted between the target picture and an I or a P picturethat is closest to the target picture on the display time axis.Therefore, prediction accuracy in motion compensation for a B picturecan be improved, resulting in improved coding efficiency.

In the above-described moving picture coding method, in the coding step,when the target picture is a B picture, a block of the target picture ispredictive-coded with reference to an already-coded B picture, andpicture position information indicating the position of the referredalready-coded B picture on the display time axis, is included in a bitstream that is obtained by coding the pictures constituting the movingpicture. Therefore, the decoding end can easily detect a referencecandidate B picture that is used as a reference picture when coding a Bpicture.

In the above-described moving picture coding method, the pictureposition information is expressed with a shorter length code as thedistance on the display time axis from the target picture to thealready-coded B picture that is referred to in coding the target pictureis shorter. Therefore, it is possible to reduce the amount of codesrequired for expressing information for identifying, at the decodingend, a candidate picture that has been forward referred to when coding aB picture.

In the above-described moving picture coding method, in the coding step,when the target picture is a B picture, a block of the target picture ispredictive-coded with reference to an already-coded B picture; andinformation indicating that the already-coded B picture is referred towhen coding the target B picture, is included as header information in abit stream that is obtained by coding the pictures constituting themoving picture. Therefore, it is easily detected at the decoding endthat another B picture is forward referred to when coding a B picture.

Further, in the present invention, there is provided a moving picturecoding method for dividing each of plural pictures constituting a movingpicture into plural blocks, and coding each picture for every block,which method includes a coding step of coding a target picture to becoded, with reference to, at least, a P picture whose block is to bepredictive-coded with reference to one already-coded picture, and a Bpicture whose block is to be predictive-coded with reference to twoalready-coded pictures; and, in the coding step, an already-codedpicture determined according to a certain rule is referred to whencoding a target block of a B picture as a target picture in a directmode which uses a motion vector of a base block that is located atspatially the same position as the target block, in an already-codedbase picture that is positioned close to the target picture. Therefore,prediction efficiency can be optimized according to the coding status.

In the above-described moving picture coding method, in the coding step,when coding the target block in the direct mode, a first already-codedpicture which is positioned immediately before the target picture and isearlier in display order than the target picture, is referred to.Therefore, prediction efficiency in coding a B picture in direct modecan be improved.

In the above-described moving picture coding method, the already-codedbase picture including the base block is a backward base P picture whichis later in display order than the target picture; and, in the codingstep, a forward motion vector (MVR×TRF/TRD) and a backward motion vector((TRB−TRD)×MVR/TRD) of the target block are obtained, on the basis of amagnitude MVR of the motion vector of the base block, a distance TRDbetween the backward base P picture and a second picture which isreferred to in coding the base block, on the display time axis, adistance TRF between the target picture and the first picture on thedisplay time axis, and a distance TRB between the target picture and thesecond picture on the display time axis, and bidirectional prediction iscarried out using the forward motion vector and the backward motionvector. Therefore, a motion vector of a target block to be used indirect mode can be accurately generated from a motion vector of a blockother than the target block.

In the above-described moving picture coding method, in the coding step,when coding the target block in the direct mode, bidirectionalprediction with the motion vector of the target block being zero iscarried out, with reference to an already-coded forward picture which ispositioned closest to the target picture and is earlier in display orderthan the target picture, and an already-coded backward picture which ispositioned closest to the target picture and is later in display orderthan the target picture. Therefore, when coding a B picture in directmode, scaling of a motion vector becomes unnecessary, resulting in areduction in processing amount.

In the above-described moving picture coding method, in the coding step,when coding the target block in the direct mode, no image information ofthe target block whose prediction error information becomes zero, intothe bit stream corresponding to the moving picture, is inserted.Therefore, the amount of codes can be reduced.

In the above-described moving picture coding method, in the coding step,when the prediction error information of the target block becomes zero,insertion of the image information of the target block into the bitstream corresponding to the moving picture, is omitted. Therefore, theamount of codes can be reduced.

In the above-described moving picture coding method, in the coding step,reference picture indices are assigned to candidate pictures for areference picture to be referred to when coding the target picture, andwhen coding the target block in the direct mode, a candidate picture towhich a specific reference picture index is assigned is referred to.Therefore, prediction efficiency can be optimized according to thecoding status.

In the above-described moving picture coding method, in the coding step,when a picture immediately before the target picture is a picture to beused as a candidate picture for a reference picture only when coding thetarget picture, the specific reference picture index is assigned to apicture which is positioned forward the target picture, except thepicture immediately before the target picture, among the candidatepictures to be referred to in coding the target picture. Therefore,prediction efficiency in coding a B picture in direct mode can beimproved.

In the above-described moving picture coding method, in the coding step,the specific reference picture index is assigned to a candidate picturewhich is earlier in display order than the target picture and is closestto the target picture, except the picture immediately before the targetpicture, among the candidate pictures to be referred to in decoding thetarget picture. Therefore, prediction efficiency in coding a B picturein direct mode can be improved.

In the above-described moving picture coding method, the already-codedbase picture including the base block is a backward base P picture whichis later in display order than the target picture; and, in the codingstep, when coding the target block in the direct mode, a first forwardpicture which is earlier in display order than the target block, whichis referred to in coding the base block, is referred to. Therefore,prediction efficiency in coding a B picture in direct mode can beimproved.

In the above-described moving picture coding method, in the coding step,when coding the target block in the direct mode, a second forwardpicture which is positioned immediately before the target picture and isearlier in display order than the target picture, is referred to; and aforward motion vector (MVR×TRF/TRD) and a backward motion vector((TRB−TRD)×MVR/TRD) of the target block are obtained, on the basis of amagnitude MVR of the motion vector of the base block, a distance TRDbetween the backward base P picture and the first forward picture on thedisplay time axis, a distance TRF between the target picture and thesecond forward picture on the display time axis, and a distance TRBbetween the target picture and the first forward picture on the displaytime axis. Therefore, a motion vector of a target block to be used indirect mode can be accurately generated from a motion vector of a blockother than the target block.

In the above-described moving picture coding method, in the coding step,when coding the target block in the direct mode, if a forward picture tobe referred to, which is earlier in display order than the targetpicture, does not exist in a memory for holding reference pictures, apicture which is closest to the target picture and is earlier in displayorder than the target picture, is referred to. Therefore, it is possibleto prevent prediction efficiency in coding a B picture in direct modefrom being significantly degraded.

Further, in the present invention, there is provided a moving picturecoding method for coding each of plural pictures constituting a movingpicture to generate a bit stream corresponding to each picture, whichmethod includes a coding step of coding a target picture to be coded,with reference to an already-coded picture; and the coding stepincludes: an index assignment step of assigning reference pictureindices to plural reference candidate pictures which are candidates fora reference picture to be referred to in coding the target picture, insuch a manner that a smaller reference picture index is assigned to acandidate picture which is closer in display order to the target pictureto be coded, and an index addition step of adding the reference pictureindex which is assigned to a picture that is referred to in coding thetarget picture, to the bit stream. Therefore, a smaller referencepicture index can be assigned to a picture which is timewise closer tothe target picture and is more likely to be selected as a referencepicture, whereby the amount of codes for the reference picture indicescan be minimized, resulting in improved coding efficiency.

Further, in the present invention, there is provided a moving picturecoding method for coding each of plural pictures constituting a movingpicture to generate a bit stream corresponding to each picture, whichmethod includes a coding step of coding a target picture to be codedwith reference to an already-coded picture and, in the coding step, aflag indicating whether or not the target picture should be used as acandidate for a reference picture when coding another picture thatfollows the target picture, is described in the bit stream. Therefore,when coding a B picture to be subjected to bidirectional predictivecoding, a forward picture that is closest to this B picture can be usedas a reference picture, whereby prediction accuracy in motioncompensation for the B picture can be increased, resulting in improvedcoding efficiency.

In the present invention, there is provided a moving picture decodingmethod for decoding each of plural pictures constituting a movingpicture, for every block that is a processing unit of each picture,thereby converting a bit stream corresponding to each picture into imagedata, which method includes a decoding step of performing predictivedecoding for a block of a target picture to be decoded, with referenceto an already-decoded picture; and, in the decoding step, when thetarget picture is a B picture whose block is to be predictive-decodedwith reference to two already-decoded pictures, a block of the targetpicture is predictive-decoded with reference to an already-decoded Bpicture. Therefore, a block of a B picture, which has been coded using aB picture as a candidate picture for forward reference, can be correctlydecoded.

In the above-described moving picture decoding method, in the decodingstep, when the target picture is a B picture, a block of the targetpicture is predictive-decoded with reference to an already-decoded Bpicture, and when the target picture is a P picture whose block is to bepredictive-decoded with reference to one already-decoded picture, eachblock of the target picture is predictive-decoded without referring toany already-decoded B picture. Therefore, in a memory where pictures tobe candidates for a reference picture are stored, management of thecandidate pictures is facilitated.

In the above-described moving picture decoding method, each of theplural pictures constituting the moving picture is, in the decodingstep, decoded as one of the following pictures: an I picture whose blockis to be decoded without referring to an already-decoded picture, a Ppicture whose block is to be predictive-decoded with reference to onealready-decoded picture, and a B picture whose block is to bepredictive-decoded with reference to two already-decoded pictures; and,in the decoding step, when the target picture is a B picture, a block ofthe target picture is predictive-decoded with reference to analready-decoded B picture, and the number of candidate pictures for aforward reference picture to be referred to when decoding the targetpicture as a B picture is equal to or smaller than the number ofcandidate pictures for a reference picture to be referred to whendecoding target picture as a P picture. Therefore, it is possible toavoid an increase in capacity of the memory holding the referencecandidate pictures, which is caused by that another B picture isreferred to when decoding a B picture.

In the above-described moving picture decoding method, each of theplural pictures constituting the moving picture is, in the decodingstep, decoded as one of the following pictures: an I picture whose blockis to be decoded without referring to an already-decoded picture, a Ppicture whose block is to be predictive-decoded with reference to onealready-decoded picture, and a B picture whose block is to bepredictive-decoded with reference to two already-decoded pictures; and,in the decoding step, when the target picture is a B picture, a Bpicture to be referred to in predictive-decoding a block of the targetpicture is only a B picture which is inserted between the target pictureand an I or a P picture that is closest to the target picture in displayorder. Therefore, prediction accuracy in motion compensation for a Bpicture can be improved.

In the above-described moving picture decoding method, in the decodingstep, when the target picture is a B picture, a process ofpredictive-decoding a block of the target picture with reference to analready-decoded B picture, is carried out on the basis of pictureposition information indicating the position of the already-decoded Bpicture on the display time axis, which information is included in thebit stream. Therefore, the decoding end can correctly detect a referencecandidate B picture that has been used as a reference picture whencoding a B picture.

In the above-described moving picture decoding method, the pictureposition information is expressed with a shorter length code as thedistance on the display time axis from the target picture to thealready-decoded forward B picture that is referred to in decoding thetarget picture is shorter. Therefore, it is possible to reduce theamount of codes required for expressing information for identifying, atthe decoding end, a candidate picture that has been forward referred towhen coding a B picture.

In the above-described moving picture decoding method, in the decodingstep, when the target picture is a B picture, a process ofpredictive-decoding a block of the target picture with reference to analready-decoded B picture, is carried out with reference to headerinformation indicating that an already-coded B picture is referred towhen coding the target B picture, which header information is includedin the bit stream corresponding to the picture as a component of themoving picture. Therefore, in predictive decoding for a target block, itcan be reliably and speedily detected that another B picture is forwardreferred to when coding a B picture.

Further, in the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture, for every block that is a processing unit of eachpicture, thereby converting a bit stream corresponding to each pictureinto image data, which method includes a decoding step of decoding atarget picture to be decoded, with reference to, at least, a P picturewhose block is to be predictive-decoded with reference to onealready-decoded picture, and a B picture whose block is to bepredictive-decoded with reference to two already-decoded pictures; and,in the decoding step, an already-decoded picture determined according toa certain rule is referred to when decoding a target block of a Bpicture as a target picture in a direct mode which uses a motion vectorof a base block that is located at spatially the same position as thetarget block, in an already-decoded base picture that is positionedclose to the target picture. Therefore, it is possible to realize adecoding method corresponding to a coding method that can optimizeprediction efficiency according to the coding status.

In the above-described moving picture decoding method, in the decodingstep, when decoding the target block in the direct mode, a firstalready-coded picture which is positioned immediately before the targetpicture and is earlier in display order than the target picture, isreferred to. Therefore, prediction efficiency in decoding a B picture indirect mode can be improved.

In the above-described moving picture decoding method, thealready-decoded base picture including the base block is a backward baseP picture which is later in display order than the target picture; and,in the decoding step, a forward motion vector (MVR×TRF/TRD) and abackward motion vector ((TRB−TRD)×MVR/TRD) of the target block areobtained, on the basis of a magnitude MVR of the motion vector of thebase block, a distance TRD between the backward base P picture and asecond picture which is referred to in decoding the base block, on thedisplay time axis, a distance TRF between the target picture and thefirst picture on the display time axis, and a distance TRB between thetarget picture and the second picture on the display time axis.Therefore, a motion vector of a target block to be used in direct modecan be accurately generated from a motion vector of a block other thanthe target block.

In the above-described moving picture decoding method, in the decodingstep, when decoding the target block in the direct mode, bidirectionalprediction with the motion vector of the target block being zero iscarried out, referring to an already-decoded forward picture which ispositioned closest to the target picture and is earlier in display orderthan the target picture as well as referring to an already-decodedbackward picture which is positioned closest to the target picture andis later in display order than the target picture. Therefore, whendecoding a B picture in direct mode, scaling of a motion vector becomesunnecessary, resulting in reduced processing amount.

In the above-described moving picture decoding method, in the decodingstep, when decoding the target block in the direct mode, imageinformation of the target block having prediction error information ofzero, which image information is not included in the bit stream, isrestored using the motion vector of the base block. Therefore, a targetblock which is not included in the bit stream and has prediction errorinformation being zero, can be predictive-decoded using a motion vectorof another block.

In the above-described moving picture decoding method, in the decodingstep, image information of the target block having prediction errorinformation of zero, which image information is not included in the bitstream, is restored using the motion vector of the base block.Therefore, a target block which is not included in the bit stream andhas prediction error information of zero, can be predictive-decodedusing a motion vector of another block.

In the above-described moving picture decoding method, reference pictureindices are assigned to candidate pictures for a reference picture to bereferred to when decoding the target picture; and, in the decoding step,when decoding the target block in the direct mode, a candidate pictureto which a specific reference picture index is assigned is referred to.Therefore, it is possible to realize a decoding method corresponding toa coding method that can optimize prediction efficiency according to thecoding status.

In the above-described moving picture decoding method, when a pictureimmediately before the target picture is a picture to be used as acandidate picture for a reference picture only when decoding the targetpicture, the specific reference picture index is assigned to a targetpicture which is positioned forward the target picture, except thepicture immediately before the target picture, among the candidatepictures to be referred to in decoding the target picture; and, in thedecoding step, when decoding the target block in the direct mode, thepicture to which the specific reference picture index is assigned isreferred to. Therefore, prediction efficiency in decoding a B picture indirect mode can be improved.

In the above-described moving picture decoding method, the specificreference picture index is assigned to a candidate picture which isearlier in display order than the target picture and is closest to thetarget picture, except the picture immediately before the targetpicture, among the candidate pictures to be referred to in decoding thetarget picture; and, in the decoding step, when decoding the targetblock in the direct mode, the picture to which the specific referencepicture index is assigned is referred to. Therefore, predictionefficiency in decoding a B picture in direct mode can be improved.

In the above-described moving picture decoding method, thealready-decoded base picture including the base block is a backward baseP picture which is later in display order than the target picture; and,in the decoding step, when decoding the target block in the direct mode,a first forward picture which is earlier in display order than thetarget block, which is referred to in decoding the base block, isreferred to. Therefore, prediction efficiency in decoding a B picture indirect mode can be improved.

In the above-described moving picture decoding method, in the decodingstep, when decoding the target block in the direct mode, a secondforward picture which is positioned immediately before the targetpicture and is earlier in display order than the target picture, isreferred to; and a forward motion vector (MV×TRF/TRD) and a backwardmotion vector ((TRB−TRD)×MVR/TRD) of the target block are obtained, onthe basis of a magnitude MVR of the motion vector of the base block, adistance TRD between the backward base P picture and the first forwardpicture on the display time axis, a distance TRF between the targetpicture and the second forward picture on the display time axis, and adistance TRB between the target picture and the first forward picture onthe display time axis. Therefore, a motion vector of a target block tobe used in direct mode can be accurately generated from a motion vectorof a block other than the target block.

In the above-described moving picture decoding method, in the decodingstep, when decoding the target block in the direct mode, if a forwardpicture to be referred to, which is earlier in display order than thetarget picture, does not exist in a memory for holding referencepictures, a picture which is closest to the target picture and isearlier in display order than the target picture is referred to.Therefore, it is possible to prevent prediction efficiency in decoding aB picture in direct mode from being significantly degraded.

Further, in the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture to convert a bit stream corresponding to each pictureinto image data, which method includes a decoding step of decoding atarget picture to be decoded, with reference to an already-decodedpicture; and the decoding step includes: an index assignment step ofassigning reference picture indices to plural reference candidatepictures which are candidates for a reference picture to be referred toin decoding the target picture, in such a manner that a smallerreference picture index is assigned to a candidate picture which iscloser in display order to the target picture to be decoded, and areference picture determination step of determining a picture to bereferred to in decoding the target picture, on the basis of a referencepicture index assigned to a picture that is referred to in coding thetarget picture, which index is added to the bit stream of the targetpicture, and the reference picture indices assigned to the referencecandidate pictures in the index assignment step. Therefore, it ispossible to correctly decode a bit stream which has been generated by ahighly efficient coding method in which a smaller reference pictureindex can be assigned to a picture that is timewise closer to the targetpicture and is more likely to be selected.

Further, in the present invention, there is provided a moving picturedecoding method for decoding each of plural pictures constituting amoving picture to convert a bit stream corresponding to each pictureinto image data, which method includes a decoding step of decoding atarget picture to be decoded with reference to an already-decodedpicture, wherein a flag indicating whether or not the target pictureshould be used as a candidate for a reference picture when decodinganother picture that follows the target picture, is described in the bitstream, and in the decoding step, management of the decoded targetpicture is carried out on the basis of the flag. Therefore, it ispossible to correctly decode a bit stream corresponding to a B picture,which is generated by using, as forward reference pictures, a B picturesubjected to bidirectional predictive coding as well as a P picturesubjected to forward predictive coding.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram for explaining a moving picture codingapparatus according to a first embodiment of the present invention.

FIGS. 2(a) and 2(b) are schematic diagrams for explaining a movingpicture coding method according to the first embodiment, wherein FIG.2(a) shows pictures arranged in display order, and FIG. 2(b) showspictures arranged in coding order.

FIG. 3 is a schematic diagram for explaining the moving picture codingapparatus according to the first embodiment and a moving picturedecoding apparatus according to a second embodiment, illustrating amethod for collectively managing P and B pictures in a memory.

FIGS. 4(a) and 4(b) are diagrams for explaining the first embodiment,illustrating a first example (4(a)) and a second example (4(b)) ofdirect mode coding (for picture B11).

FIGS. 5(a) and 5(b) are diagrams for explaining the first embodiment,illustrating a third example (5(a)) and a fourth example (5(b)) ofdirect mode coding (for picture B11).

FIGS. 6(a)-6(c) are diagrams for explaining the first embodiment,illustrating a fifth example (6(a)) of direct mode coding (for pictureB11), a skip block (6(b)), and a skip identifier (6(c)).

FIGS. 7(a) and 7(b) are diagrams for explaining the first embodiment,illustrating a first example (7(a)) and a second example (7(b)) ofdirect mode coding (for picture B12).

FIGS. 8(a) and 8(b) are diagrams for explaining the first embodiment,illustrating a third example (8(a)) and a fourth example (8(b)) ofdirect mode coding (for picture B12).

FIGS. 9(a) and 9(b) are diagrams for explaining the first embodiment,illustrating a first example (9(a)) and a second example (9(b)) ofcoding wherein a B picture positioned forward a closest forward Ppicture is referred to.

FIGS. 10(a) and 10(b) are diagrams for explaining the first embodiment,illustrating a first example (10(a)) and a second example (10(b)) ofcoding wherein a B picture positioned forward a closest forward I or Ppicture is not referred to.

FIG. 11 is a diagram for explaining the first and second embodiments,illustrating a first method for managing P pictures and B pictures in amemory, separately from each other.

FIG. 12 is a diagram for explaining the first and second embodiments,illustrating a second method for managing P pictures and B pictures in amemory, separately from each other.

FIG. 13 is a diagram for explaining the first and second embodiments,illustrating a third method for managing P pictures and B pictures in amemory, separately from each other.

FIG. 14 is a diagram for explaining the first and second embodiments,illustrating a fourth method for managing P pictures and B pictures in amemory, separately from each other.

FIG. 15 is a block diagram for explaining a moving picture decodingapparatus according to the second embodiment of the invention.

FIGS. 16(a) and 16(b) are schematic diagrams for explaining a movingpicture decoding method according to the second embodiment, wherein FIG.16(a) shows pictures arranged in decoding order, and FIG. 16(b) showspictures arranged in display order.

FIG. 17 is a diagram for explaining the second embodiment, illustratingbidirectional predictive decoding (for picture B11).

FIGS. 18(a) and 18(b) are diagrams for explaining the second embodiment,illustrating a first example (18(a)) and a second example (18(b)) ofdirect mode decoding (for picture B11).

FIGS. 19(a) and 19(b) are diagrams for explaining the second embodiment,illustrating a third example (19(a)) and a fourth example (19(b)) ofdirect mode decoding (for picture B11).

FIG. 20 is a diagram for explaining the second embodiment, illustratingbidirectional predictive decoding (for picture B12).

FIGS. 21(a) and 21(b) are diagrams for explaining the second embodiment,illustrating a first example (21(a)) and a second example (21(b)) ofdirect mode decoding (for picture B12).

FIGS. 22(a) and 22(b) are diagrams for explaining the second embodiment,illustrating a third example (22(a)) and a fourth example (22(b)) ofdirect mode decoding (for picture B12).

FIG. 23 is a block diagram for explaining a moving picture codingapparatus according to a third embodiment of the invention.

FIG. 24 is a schematic diagram for explaining the moving picture codingapparatus according to the third embodiment, illustrating a method forcollectively managing P and B pictures in a memory.

FIGS. 25(a) and 25(b) are diagrams for explaining the third embodiment,illustrating a case where decoding of a B picture immediately after a Ppicture is not carried out (25(a)), and a case where a predeterminedpicture is not decoded.

FIG. 26 is a block diagram for explaining a moving picture decodingapparatus according to a fourth embodiment of the invention.

FIG. 27 is a block diagram for explaining a moving picture codingapparatus according to a fifth embodiment of the invention.

FIG. 28 is a diagram for explaining the fifth embodiment, illustrating amethod for managing a picture memory, and a method for assigningreference picture indices.

FIGS. 29(a) and 29(b) are diagrams for explaining the fifth embodiment,illustrating pictures arranged in display order (29(a)), and picturesarranged in coding order.

FIG. 30 is a diagram for explaining the fifth embodiment, illustrating amethod for managing a picture memory, and a method for assigningreference picture indices.

FIG. 31 is a diagram for explaining the fifth embodiment, illustrating adata structure of a bit stream corresponding to a block in a case wheretwo systems of reference picture indices are used.

FIG. 32 is a block diagram for explaining a moving picture decodingapparatus according to a sixth embodiment of the present invention.

FIG. 33 is a block diagram for explaining a moving picture codingapparatus according to a seventh embodiment of the invention.

FIGS. 34(a) and 34(b) are schematic diagrams for explaining a movingpicture coding method according to the seventh embodiment, illustratingpictures arranged in display order (34(a)), and pictures arranged incoding order (34(b)).

FIG. 35 is a block diagram for explaining a moving picture decodingapparatus according to an eighth embodiment of the invention.

FIGS. 36(a) and 36(b) are schematic diagrams for explaining a movingpicture decoding method according to the eighth embodiment, illustratingpictures arranged in decoding order (36(a)), and pictures arranged indisplay order (36(b)).

FIG. 37 is a diagram for explaining the eighth embodiment, illustratinga method for managing a picture memory.

FIGS. 38(a) and 38(b) are diagrams illustrating a storage medium whichcontains a program for implementing the apparatuses according to therespective embodiments with software, and FIG. 38(c) is a diagramillustrating a computer system using the storage medium.

FIG. 39 is a diagram for explaining applications of the moving picturecoding methods and decoding methods according to the respectiveembodiments, illustrating a contents supply system which realizescontents distribution services.

FIG. 40 is a diagram for explaining a portable phone utilizing themoving picture coding methods and decoding methods according to therespective embodiments.

FIG. 41 is a block diagram illustrating a specific construction of theportable phone shown in FIG. 40.

FIG. 42 is a conceptual diagram illustrating a system for digitalbroadcasting that utilizes the moving picture coding methods anddecoding methods according to the respective embodiments.

FIGS. 43(a)-43(c) are diagrams for explaining a conventional movingpicture coding method, illustrating an arrangement of picturesconstituting a moving picture (43(a)), a slice obtained by dividing apicture (43(b)), and a macroblock (43(c)).

FIG. 44 is a diagram for explaining coded data of an ordinary movingpicture, illustrating structures of streams which are obtained by codingpictures constituting a moving picture.

FIG. 45 is a diagram for explaining a conventional moving picture codingmethod such as MPEG, illustrating the relationships between targetpictures and pictures to be referred to when coding the target pictures.

FIGS. 46(a) and 46(b) are diagrams for explaining conventional directmode coding, illustrating motion vectors used in the direct mode(46(a)), and relative positions of pictures to blocks (46(b)).

FIGS. 47(a) and 47(b) are diagrams for explaining a conventional methodfor assigning reference picture indices, illustrating reference indicesto be assigned to candidate pictures which are referred to when coding Ppictures and B pictures, respectively.

FIGS. 48(a) and 48(b) are diagrams for explaining a conventional movingpicture coding method, illustrating pictures constituting a movingpicture which are arranged in display order (48(a)), and those picturesarranged in coding order (48(b)).

FIG. 49 is a diagram for explaining a conventional moving picture codingmethod, illustrating an example of management of a frame memory for thepictures arranged in coding order.

FIGS. 50(a) and 50(b) are diagram for explaining problems of theconventional inter-picture predictive coding method, illustrating a casewhere bidirectional reference is carried out (50(a)), and a case wheretwo pictures are backward referred to (50(b)).

FIGS. 51(a) and 51(b) are diagrams for explaining problems of theconventional method of assigning reference picture indices, illustratingpictures arranged in display order (51(a)), and pictures arranged incoding order (51(b)).

BEST MODE TO EXECUTE THE INVENTION Embodiment 1

FIG. 1 is a block diagram for explaining a moving picture codingapparatus 10 according to a first embodiment of the present invention.

The moving picture coding apparatus 10 according to this firstembodiment divides each of plural pictures constituting a moving pictureinto predetermined data processing units (blocks), and encodes imagedata of each picture for every block.

To be specific, the moving picture coding apparatus 10 includes an inputpicture memory (hereinafter also referred to as a frame memory) 101 forholding image data (input data) Id of inputted pictures, and outputtingthe stored data Id for every block; a difference calculation unit 102for calculating difference data between image data Md of a target blockto be coded, which is outputted from the frame memory 101, andprediction data Pd of the target block, as prediction error data PEd ofthe target block; and a prediction error coding unit 103 forcompressively coding the image data Md of the target block or theprediction error data PEd. In the frame memory 101, a process ofrearranging the image data of the respective pictures inputted indisplay order to those in picture coding order is carried out on thebasis of the relationship between each target picture and a picture tobe referred to (reference picture) in predictive coding of the targetpicture.

The moving picture coding apparatus 10 further includes a predictionerror decoding unit 105 for expandingly decoding the output data (codeddata) Ed from the prediction error coding unit 103 to output decodeddifference data PDd of the target block; an addition unit 106 for addingthe decoded difference data PDd of the target block and the predictiondata Pd of the target block to output decoded data Dd of the targetblock; and a reference picture memory (hereinafter also referred to as aframe memory) 117 for holding the decoded data Dd according to a memorycontrol signal Cd2, and outputting the stored decoded data Dd as data Rdof candidates (candidate pictures) of pictures to be referred to whencoding the target block.

The moving picture coding apparatus 10 further includes a motion vectordetection unit 108 for detecting a motion vector MV of the target blockon the basis of the output data (image data of the target block) Md fromthe frame memory 101 and the output data (candidate picture data) Rffrom the frame memory 117; and a mode selection unit 109 for determininga coding mode suited to the target block on the basis of the motionvector MV of the target block and the output data Md and Rd from therespective frame memories 101 and 117, and outputting a switch controlsignal Cs2. The motion vector detection unit 108 performs motiondetection for detecting the above-mentioned motion vector with referenceto plural candidate pictures that can be referred to in predictivecoding for the target block. Further, the mode selection unit 109selects one coding mode for the target block from among plural codingmodes, which coding mode provides optimum coding efficiency. Wheninter-picture predictive coding is selected, an optimum picture isselected from among the plural candidate pictures that can be referredto.

In the moving picture coding apparatus 10 according to the firstembodiment, for a P picture (i.e., a picture for which one already-codedpicture is referred to when one block in this picture is subjected topredictive coding), one of the following coding modes is selected:intra-picture coding mode, inter-picture predictive coding mode using amotion vector, and inter-picture predictive coding mode using no motionvector (i.e., motion vector is regarded as 0). Further, for a B picture(i.e., a picture for which two already-coded pictures are referred towhen one block in this picture is subjected to predictive coding), oneof the following coding modes is selected: intra-picture coding mode,inter-picture predictive coding mode using a forward motion vector,inter-picture predictive coding mode using a backward motion picture,inter-picture predictive coding mode using bidirectional motion vectors,and direct mode. Further, in this first embodiment, when a block in theB picture is coded in the direct mode, an already coded picture that ispositioned just previous to the target picture on the display time axisis referred to.

Furthermore, the moving picture coding apparatus 10 includes a selectionswitch 111 placed between the frame memory 101 and the differencecalculation unit 102; a selection switch 112 placed between thedifference calculation unit 102 and the prediction error coding unit103; an ON/OFF switch 113 placed between the frame memory 101, and themode selection unit 109 and the motion vector detection unit 108; anON/OFF switch 114 placed between the mode selection unit 109 and theaddition unit 106; and an ON/OFF switch 115 placed between theprediction error coding unit 103 and the prediction error decoding unit105.

Moreover, the moving picture coding apparatus 10 includes a codingcontrol unit 110 for controlling ON/OFF operations of the switches113˜115 according to a switch control signal Cs1, and outputting a codegeneration control signal Cd1 and a memory control signal Cd2; and a bitstream generation unit 104 for performing variable-length coding for theoutput data (coded data) Ed from the prediction error coding unit 103 onthe basis of the code generation control signal Cd1 to output a bitstream Bs corresponding to the target block. The bit stream generationunit 104 is supplied with the motion vector MV detected by the motionvector detection unit 108 and information indicating the coding mode Msdetermined by the mode selection unit 109. The bit stream Bscorresponding to the target bock includes the motion vector MVcorresponding to the target block, and the information indicating thecoding mode Ms.

The selection switch 111 has an input terminal Ta and two outputterminals Tb1 and Tb2, and the input terminal Ta is connected to one ofthe output terminals Tb1 and Tb2 according to the switch control signalCs2. The selection switch 112 has two input terminals Tc1 and Tc2 and anoutput terminal Td, and the output terminal Td is connected to one ofthe input terminals Tc1 and Tc2 according to the switch control signalCs2. Further, in the selection switch 111, the image data Md outputtedfrom the frame memory 101 is applied to the input terminal Ta, and theimage data Md is output through one output terminal Tb1 to the inputterminal Tc1 of the selection switch 112 while the image data Md isoutput through the other output terminal Tb2 to the differencecalculation unit 102. In the selection switch 112, the image data Mdfrom the frame memory 101 is applied to one input terminal Tc1 while thedifference data PEd obtained in the difference calculation unit 102 isapplied to the other input terminal Tc2, and either the image data Md orthe difference data PEd is output through the output terminal Td to theprediction error coding unit 103.

Next, the operation will be described.

In the following descriptions of the respective embodiments, a picture(forward picture) whose display time is earlier than that of a pictureto be coded (target picture) is referred to as a picture which ispositioned timewise forward the target picture, or simply as a picturewhich is positioned forward the target picture. Further, a picture(backward picture) whose display time is later than that of the targetpicture is referred to as a picture which is positioned timewisebackward the target picture, or simply as a picture which is positionedbackward the target picture. Furthermore, in the following descriptionsof the respective embodiments, “timewise” means “in order of displaytimes” unless otherwise specified.

In the moving picture coding apparatus 10 according to the firstembodiment, the input image data Id is input to the frame memory 101 inunits of pictures according to order of display times.

FIG. 2(a) is a diagram for explaining the order in which the image dataof the respective pictures are stored in the frame memory 101. In FIG.2(a), vertical lines indicate pictures. As for a symbol at the lowerright side of each picture, the first letter of alphabet indicates apicture type (I, P, or B), and the following numeral indicates a picturenumber in time order. That is, pictures P1, B2, B3, P4, B5, B6, P7, B8,B9, P10, B11, B12, P13, B14, B15, and P16 shown in FIG. 2(a) correspondto the pictures F(k+3)˜F(k+18) [k=−2] shown in FIG. 45, and thesepictures are arranged in display order, i.e., from one having earlierdisplay time along the display time axis X.

The image data of the pictures are stored in the frame memory 101 inorder of picture display. The image data of the pictures stored in theframe memory 101, which are arranged in order of picture display, arerearranged in order of picture coding. Hereinafter, for simplification,the image data of each picture is simply referred to as a picture.

To be specific, the process of rearranging the pictures stored in theframe memory 101 from input order (display order) to coding order iscarried out on the basis of the relationships between target picturesand reference pictures in inter-picture predictive coding. That is, thisrearrangement process is carried out such that a second picture to beused as a reference picture when coding a first picture is coded priorto the first picture.

When coding a P picture, three pictures (I or P pictures) which arepositioned close to and timewise forward the target picture to be coded(P picture) are used as candidate pictures for a reference picture. Inpredictive coding for a block in the P picture, one of the threecandidate pictures at maximum is referred to.

Further, when coding a B picture, two pictures (I or P pictures) whichare positioned close to and timewise forward the target picture (Bpicture), a B picture which is positioned closest to and timewiseforward the target picture, and an I or P picture which is positionedtimewise backward the target picture, are used as candidate pictures fora reference picture. In predictive coding for a block in the B picture,two of the four candidate pictures at maximum is referred to.

To be specific, the correspondences between the pictures P10, B11, B12,and P13, and the candidate pictures for reference pictures correspondingto the respective pictures are shown by arrows in FIG. 2(a). That is,when coding the P picture P10, the pictures P1, P4, and P7 are used ascandidate pictures for a reference picture. When coding the P pictureP13, the pictures P4, P7, and P10 are used as candidate pictures for areference picture. Further, when coding the B picture B11, the picturesP7, B9, P10, and P13 are used as candidate pictures for a referencepicture. When coding the B picture B12, the pictures P7, P10, B11, andP13 are used as candidate pictures for a reference picture.

FIG. 2(b) shows the pictures in coding order, which are obtained byrearranging the pictures P1˜P16 shown in FIG. 2(a) from display order tocoding order. After the rearrangement, as shown in FIG. 2(b), thepictures shown in FIG. 2(a) are arranged from one having earlier codingtime on the time axis Y indicating the coding times (coding time axis),i.e., the pictures are arranged in order of P4, B2, B3, P7, B5, B6, P10,B8, B9, P13, B11, B12, and P16.

The data of the pictures rearranged in the frame memory 101 aresuccessively read out, for each predetermined data processing unit, fromone having earlier coding time. In this first embodiment, thepredetermined data processing unit is a data unit for which motioncompensation is carried out and, more specifically, it is a rectangleimage space (macroblock) in which 16 pixels are arranged in both thehorizontal direction and the vertical direction. In the followingdescription, a macroblock is also referred to simply as a block.

Hereinafter, the coding processes for the pictures P13, B11, and B12will be described in this order.

Coding Process for Picture P13

Initially, the coding process for the picture P13 will be described.

Since the picture P13 to be coded (target picture) is a P picture, asinter-picture predictive coding for a target block in the picture P13,one-directional inter-picture predictive coding in which onealready-coded picture that is positioned timewise forward or backwardthe target picture is referred to is carried out.

In the following description, a P picture that is positioned forward thetarget picture is used as a reference picture.

In this case, inter-picture predictive coding using forward reference iscarried out as a coding process for the picture P13. Further, B picturesare not used as reference pictures in coding P pictures. Accordingly,three forward I or P pictures are used as candidate pictures for areference picture, more specifically, the pictures P4, P7, and P10 areused. Coding of these candidate pictures has already been completed whenthe target picture is coded, and the data (decoded data) Ddcorresponding to the candidate pictures are stored in the frame memory101.

When coding a P picture, the coding control unit 110 controls therespective switches with the switch control signal Cs1 so that theswitches 113, 114, and 115 are in their ON states. The image data Mdcorresponding to the macroblock in the picture P13, which is read fromthe frame memory 101, is input to the motion vector detection unit 108,the mode selection unit 109, and the difference calculation unit 102.

The motion vector detection unit 108 detects the motion vector MV of themacroblock in the picture P13, using the coded image data Rd of thepictures P4, P7, and P10 stored in the frame memory 117. Then, thedetected motion vector MV is output to the mode selection unit 109. Themode selection unit 109 determines a coding mode for the block in thepicture P13, using the motion vector detected by the motion vectordetection unit 108. The coding mode indicates a method for coding theblock. For example, in the case of coding a P picture, as describedabove, a coding mode is selected from among the intra-picture coding,the inter-picture predictive coding using a motion vector, and theinter-picture predictive coding using no motion vector (i.e., motion isregarded as 0). In determining a coding mode, generally, a coding modewhich minimizes coding error when a predetermined amount of bits isgiven to the block as an amount of codes, is selected. In this case,when inter-picture predictive coding is selected, a most suitablepicture is selected as a reference picture from among the pictures P4,P7, and P10.

The coding mode Ms determined by the mode selection unit 109 is outputto the bit stream generation unit 104. Further, when the determinedcoding mode Ms is the coding mode which refers to a forward picture, avector (forward motion vector) MVp that is obtained by motion detectionwith reference to the forward picture as well as information Rpindicating which one of the pictures P4, P7, and P10 is referred to whendetecting the motion vector, are also output to the bit streamgeneration unit 104.

When the coding mode Ms determined by the mode selection unit 109 is theinter-picture predictive coding mode, the motion vector MVp to be usedin the inter-picture predictive coding, and information Rp indicatingwhich one of the pictures P4, P7, and P10 is referred to when detectingthe motion vector, are stored in the motion vector storage unit 116.

Further, the mode selection unit 109 performs motion compensationaccording to the coding mode determined for the target block, using themotion vectors corresponding to the reference picture and the targetblock. Then, prediction data Pd for the target block, which is obtainedby the motion compensation, is output to the difference calculation unit102 and the addition unit 106.

However, when the intra-picture coding mode is selected, the modeselection unit 109 does not generate prediction data Pd. Further, whenthe intra-picture coding mode is selected, the switch 111 is controlledso that the input terminal Ta is connected to the output terminal Tb1,and the switch 112 is controlled so that the output terminal Td isconnected to the input terminal Tc1. On the other hand, when theinter-picture predictive coding is selected, the switch 111 iscontrolled so that the input terminal Ta is connected to the outputterminal Tb2, and the switch 112 is controlled so that the outputterminal Td is connected to the input terminal Tc2.

Hereinafter, a description will be given of a case where theinter-picture predictive coding mode is selected as the coding mode Ms.

The difference calculation unit 102 is supplied with the image data Mdof the target block in the picture P13, and the corresponding predictiondata Pd from the mode selection unit 109. The difference calculationunit 102 calculates difference data between the image data of the blockin the picture P13 and the corresponding predictive data Pd, and outputsthe difference data as prediction error data PEd.

The prediction error data PEd is input to the prediction error codingunit 103. The prediction error coding unit 103 subjects the inputtedprediction error data PEd to coding processes such as frequencyconversion and quantization to generate coded data Ed. The processessuch as frequency conversion and quantization are carried out in unitsof data corresponding to a rectangle image space (sub-block) in whicheight pixels are arranged in both the horizontal direction and thevertical direction.

The coded data Ed outputted from the prediction error coding unit 103 isinput to the bit stream generation unit 104 and the prediction errordecoding unit 105.

The bit stream generation unit 104 generates a bit stream by subjectingthe inputted coded data Ed to variable-length coding. Further, the bitstream generation unit 104 adds, to the bit stream, information such asthe motion vector MVp and the coding mode Ms, header informationsupplied from the coding control unit 110, and the like, therebygenerating a bit stream Bs.

When the coding mode is one performing forward reference, information(reference picture information) Rp indicating which one of the picturesP4, P7, and P10 is referred to when detecting the forward motion vectoris also added to the bit stream.

Next, a description will be given of a method for managing the framememory, and information indicating a reference picture among candidatepictures (reference picture information).

FIG. 3 is a diagram illustrating how the pictures stored in thereference picture memory (frame memory) 117 change with time. Managementof this frame memory 117 is carried out according to the memory controlsignal Cd2 from the coding control unit 110. Further, the frame memory117 has memory areas (#1)˜(#5) for five pictures. Each memory area canhold image data corresponding to one picture. However, each memory areais not necessarily an area in one frame memory, it may be one memory.

Initially, a method for managing the frame memory (reference picturememory) will be described.

When coding of a picture P13 is started, pictures B8, P4, P7, P10, andB9 are stored in the respective memory areas (#1)˜(#5) in the framememory 117, respectively. Although the picture B9 is not used for codingof the picture P13, the picture B9 is stored in the frame memory 117because it is used for coding of the picture B11. The picture P13 iscoded using the pictures P4, P7, and P10 as candidate pictures for areference picture. The coded picture P13 is stored in the memory area(#1) where the picture P8 had been stored. The reason is as follows.Although the pictures P4, P7, P10, and B9 are used as candidate picturesfor a reference picture when coding the picture P13 and the followingpictures, the picture B8 is not used as a reference picture when codingthese pictures. In FIG. 3, each circled picture is a picture (targetpicture) which is finally stored in the frame memory 117 when coding ofthe target picture has completed.

Next, a description will be given of a method for assigning a specificreference picture index as reference picture information, to eachcandidate picture.

The reference picture index is information indicating which one ofplural candidate pictures for a reference picture is used as a referencepicture when coding each block. In other words, the reference pictureindex is information indicating which one of the candidate pictures P4,P7, and P10 for a reference picture is used when detecting the motionvector of the target block in the target picture (picture P13). As forassignment of reference picture indices, a method of successivelyassigning the indices to the respective candidate pictures, startingfrom a candidate picture that is timewise closest to the target picture.

To be specific, when the picture P10 is designated as a referencepicture in coding the target block in the target picture p13,information indicating that a candidate picture just previous to thetarget picture P13 is designated as a reference picture (referencepicture index [0]) is added into the bit stream of the target pictureP13. When the picture P7 is referred to in coding the block in thetarget picture P13, information indicating that a candidate picturetwo-pictures previous to the target picture P13 is designated as areference picture (reference picture index [1]) is added into the bitstream of the target picture P13. When the picture P4 is referred to incoding the block in the target picture P13, information indicating thata candidate picture three-pictures previous to the target picture P13 isdesignated as a reference picture (reference picture index [2]) is addedinto the bit stream of the target picture P13.

In FIG. 3, a picture that is assigned a code [b] as reference pictureinformation will be a candidate for a backward reference picture whencoding the target picture.

Coding Process for Picture B11

Next, the coding process for the picture B11 will be described.

Since the picture to be coded (target picture) is the picture B11,inter-picture predictive coding to be performed for a target block inthe picture B11 is bidirectional inter-picture predictive coding inwhich two already-coded pictures that are timewise forward or backwardthe target picture are referred to.

Hereinafter, a description will be given of a case where one picture (Ipicture, P picture or B picture) positioned forward the target pictureand one picture (I picture or P picture) positioned backward the targetpicture are used as reference pictures.

That is, in this case, as forward reference pictures, two pictures (I orP pictures) positioned timewise close to the target picture (pictureB11) and a B picture positioned timewise closest to the target pictureare used. Further, as a backward reference picture, an I or P picturepositioned timewise closest to the target picture is used. Accordingly,in this case, candidate pictures for a reference picture for the pictureB11 are the pictures P7, B9, and P10 (forward pictures) and the pictureP 13 (backward picture).

In coding a B picture to be used as a reference picture when codinganother picture, the coding control unit 110 controls the respectiveswitches with the switch control signal Cs1 so that the switches 113,114, and 115 are turned ON. Since the picture B11 is to be used as areference picture when coding another picture, the coding control unit110 controls the respective switches with the switch control signal Cs2so that the switches 113, 114, and 115 are turned ON. The image data Mdcorresponding to the block in the picture B11, which is read from theframe memory 101, is input to the motion vector detection unit 108, themode selection unit 109, and the difference calculation unit 102.

The motion vector detection unit 108 detects a forward motion vector anda backward motion vector of the target block in the picture B11. Indetecting these motion vectors, the pictures P7, B9, and P10 stored inthe frame memory 117 are used as forward reference pictures, and thepicture P13 is used as a backward reference picture. Detection of abackward motion vector is carried out based on the picture P13 as abackward reference picture. The motion vectors detected by the motionvector detection unit 108 are output to the mode selection unit 109.

The mode selection unit 109 determines a coding mode for the targetblock in the picture B11, using the motion vectors detected by themotion vector detection unit 108. For example, a coding mode for the Bpicture B11 is selected from among the intra-picture coding mode, theinter-picture predictive coding mode using a forward motion vector, theinter-picture predictive coding mode using a backward motion picture,the inter-picture predictive coding mode using bidirectional motionvectors, and the direct mode. When the coding mode is inter-picturepredictive coding using a forward motion vector, a most suitable pictureis selected as a reference picture from among the pictures P7, B9, andP10.

Hereinafter, a process of coding the blocks in the picture B11 by thedirect mode will be described.

First Example of Direct Mode Coding

FIG. 4(a) shows a first example of direct mode coding for a block(target block) BLa1 in the picture (target picture) B11. This directmode coding utilizes a motion vector (base motion vector) MVc1 of ablock (base block) BLb1 which is included in the picture (base picture)P13 as a reference picture positioned backward the picture B11 and islocated in the same position as the target block BLa1. The motion vectorMVc1 is a motion vector which is used when coding the block BLb1 in thepicture P13, and it is stored in the motion vector storage unit 116.This motion vector MVc1 is detected with reference to the picture P10,and indicates an area CRc1 in the picture P10, which area corresponds tothe block BLb1. The block BLa1 is subjected to bidirectional predictivecoding, using motion vectors MVd1 and MVe1 which are parallel to themotion vector MVc1, and the pictures P10 and P13 which are selected asreference pictures. The motion vectors MVd1 and MVe1 which are used incoding the block BLa1 are a forward motion vector indicating an areaCRd1 in the picture P10, corresponding to the block BLa1, and a backwardmotion vector indicating an area CRe1 in the picture P13, correspondingto the block BLa1, respectively.

In this case, the magnitude MVF of the forward motion vector MVd1 andthe magnitude MVB of the backward motion vector MVe1 are obtained byformulae (1) and (2) as follows.

MVF=MVR×TRF/TRD  (1)

MVB=(TRB−TRD)×MVR/TRD  (2)

where MVF and MVB represent the horizontal component and the verticalcomponent of the motion vectors, respectively.

Further, MVR is the magnitude of the motion vector MVc1 (a direction ona two-dimensional space is expressed by a sign), and TRD is thetime-basis distance between the backward reference picture (picture P13)for the target picture (picture B11) and the picture P10 which isreferred to when coding the block BLb1 in the backward reference picture(picture P13). Further, TRF is the time-basis distance between thetarget picture (picture B11) and the just-previous reference picture(picture P10), and TRB is the time-basis distance between the targetpicture (picture B11) and the picture P10 which is referred to whencoding the block BLb1 in the backward reference picture (picture P13).

Second Example of Direct Mode Coding

Next, a second example of direct mode coding will be described.

FIG. 4(b) shows a second example of a process for coding a block (targetblock) BLa2 in the picture (target picture) B11 by the direct mode.

This direct mode coding utilizes a motion vector (base motion vector)MVf2 of a block (base block) BLb2 which is included in the picture (basepicture) P13 as a reference picture positioned backward the picture B11and is located in the same position as the target block BLa2. The motionvector MVf2 is a motion vector which is used when coding the block BLb2,and it is stored in the motion vector storage unit 116. This motionvector MVf2 is detected with reference to the picture P7, and indicatesan area CRf2 in the picture P7, which area corresponds to the blockBLb2. The block BLa2 is subjected to bidirectional predictive coding,using motion vectors MVg2 and MVh2 which are parallel to the motionvector MVf2, and the pictures P10 and P13 which are selected asreference pictures. The motion vectors MVg2 and MVh2 which are used incoding the block BLa2 are a forward motion vector indicating an areaCRg2 in the picture P10, corresponding to the block BLa2, and a backwardmotion vector indicating an area CRh2 in the picture P13, correspondingto the block BLa2, respectively.

In this case, the magnitudes MVF and MVB of the motion vectors MVg2 andMVh2 are obtained by the above-described formulae (1) and (2),respectively.

As described above, in the direct mode, the motion vector MVf2 of theblock BLb2, which is included in the picture to be used as a backwardreference picture when coding the target block BLa2 and is located inrelatively the same position as the target block, is scaled, therebyobtaining the forward motion vector MVg2 and the backward motion vectorMVh2 for the target block. Therefore, when the direct mode is selected,it is not necessary to send information of the motion vector of thetarget block. Furthermore, since the already-coded picture which ispositioned timewise closest to the target picture is used as a forwardreference picture, prediction efficiency can be improved.

Third Example of Direct Mode Coding

Next, a third example of direct mode coding will be described.

FIG. 5(a) shows a third example of a process of coding a block (targetblock) BLa3 in the picture (target picture) B11 by the direct mode.

This direct mode coding utilizes a motion vector (base motion vector)MVc3 of a block (base block) BLb3 which is included in the picture (basepicture) P13 that is a backward reference picture for the picture B11and is located in the same position as the target block BLa3. The motionvector MVc3 is a motion vector which is used when coding the block BLb3,and it is stored in the motion vector storage unit 116. This motionvector MVc3 is detected with reference to the picture P7, and indicatesan area CRc3 in the picture P7, which area corresponds to the blockBLb3. The block BLa3 is subjected to bidirectional predictive coding, onthe basis of motion vectors MVd3 and MVe3 which are parallel to themotion vector MVc3, the picture which is referred to when coding theblock BLb3 (the picture P7 selected as a forward reference picture), andthe picture P13 as a backward reference picture. In this case, themotion vectors MVd3 and MVe3 which are used in coding the block BLa3 area forward motion vector indicating an area CRd3 in the picture P7,corresponding to the block BLa3, and a backward motion vector indicatingan area CRe3 in the picture P13, corresponding to the block BLa3,respectively.

The magnitudes MVF and MVB of the motion vectors MVd3 and MVe3 areobtained by the following formula (3) and the above-described formula(2), respectively.

MVF=MVR×TRB/TRD  (3)

where MVR is the magnitude of the motion vector MVc3.

As described above, in the direct mode coding shown in FIG. 5(a), themotion vector MVc3 of the block BLb3, which is included in the pictureto be used as a backward reference picture when coding the target blockand is located in relatively the same position as the target block, isscaled, thereby obtaining the forward motion vector MVd3 and thebackward motion vector MVe4 for the target block. Therefore, when thedirect mode is selected, it is not necessary to send information of themotion vector of the target block.

When the picture P13 to be referred to in coding the block BLb3 hasalready been deleted from the frame memory 117, the forward referencepicture P10 that is timewise closest to the target picture is used as aforward reference picture in the direct mode. The direct mode coding inthis case is identical to that shown in FIG. 4(a) (first example).

Fourth Example of Direct Mode Coding

Next, a fourth example of direct mode coding will be described.

FIG. 5(b) shows a fourth example of a process of coding a block (targetblock) BLa4 in the picture (target picture) B11 by the direct mode.

In this case, the target block BLa4 is subjected to bidirectionalpredictive coding with a motion vector being 0, on the basis of theclosest picture P10 that is selected as a forward reference picture, andthe picture P13 as a backward reference picture. That is, motion vectorsMVf4 and MVh4 to be used for coding the block BLa4 are a motion vectorindicating an area (block) CRf4 that is included in the picture P10 andis located in relatively the same position as the target block BLa4, anda motion vector indicating an area (block) CRh4 that is included in thepicture P13 and is located in relatively the same position as the targetblock BLa4, respectively.

As described above, in the direct mode coding shown in FIG. 5(b), sincethe motion vector of the target block is forcefully set to 0, when thedirect mode is selected, it is not necessary to send information of themotion vector of the target block, and scaling of the motion vectorbecomes unnecessary, resulting in a reduction in complexity of signalprocessing. This method is applicable to, for example, a case where ablock which is included in the picture P13 as a backward referencepicture of the picture B11 and is located in the same position as theblock BLa4 is a block having no motion vector like an intra-frame-codedblock. Accordingly, even when a block which is included in the backwardreference picture and is located in the same position as the targetblock is coded without a motion vector, coding efficiency can beenhanced using the direct mode.

The above-mentioned direct mode processing (first to fourth examples) isapplicable not only when the interval of picture display times isconstant but also when the interval of picture display times isvariable.

Fifth Example of Direct Mode Coding

Next, direct mode predictive coding to be performed when the interval ofpicture display times is variable will be described as a fifth exampleof direct mode coding.

FIG. 6(a) is a diagram for explaining a fifth example of a direct modecoding, wherein the direct mode predictive coding (second example) isapplied to the case where the picture display interval is variable.

In this case, bidirectional predictive coding for a target block BLa5 inthe target picture B11 is carried out by using a motion vector (basemotion vector) MVf5 of a block (base block) BLb5 which is included inthe picture (base picture) P13 as a reference picture positionedbackward the picture B11 and is located in the same position as thetarget block BLa5, in the same manner as the direct mode predictivecoding (second example) shown in FIG. 4(b). The motion vector MVf5 is amotion vector which is used when coding the block BLb5 in the pictureP13, and it indicates an area CRf5 in the picture P7, which areacorresponds to the block BLb5. Further, motion vectors MVg5 and MVh5corresponding to the target block are parallel to the motion vectorMVf5. Further, these motion vectors MVg5 and MVh5 are a forward motionvector indicating an area CRg5 in the picture P10, corresponding to theblock BLa5, and a backward motion vector indicating an area CRh5 in thepicture P13, corresponding to the block BLa5, respectively.

Also in this case, the magnitudes MVF and MVB of the motion vectors MVg5and MVh5 can be obtained by the above-described formulae (1) and (2),respectively, as in the direct mode processing of the second example.

[Process of Skipping Specific Block]

Next, a description will be given of direct mode coding where a specificblock is treated as a skip block.

When difference data corresponding to a target block becomes zero indirect mode coding, the prediction error coding unit 103 does notgenerate coded data corresponding to the target block, and the bitstream generation unit 104 does not output a bit stream corresponding tothe target block. Thus, a block whose difference data becomes zero istreated as a skip block.

Hereinafter, a case where a specific block is treated as a skip blockwill be described.

FIG. 6(b) shows a specific picture F as a component of a moving picture.

In this picture F, among adjacent blocks MB(r−1)˜MB(r+3), the values ofdifference data (prediction error data) corresponding to blocks MB(r−1),MB(r), and MB(r+3) are non-zero, but the values of difference data(prediction error data) corresponding to blocks MB(r+1) and MB(r+2)which are positioned between the block MB(r) and the block MB(r+3), arezero.

In this case, the blocks MB(r+1) and MB(r+2) are treated as skip blocksin the direct mode, and a bit stream Bs corresponding to a movingpicture does not include bit streams corresponding to the blocks MB(r+1)and MB(r+2).

FIG. 6(c) is a diagram for explaining a stream structure in the casewhere the blocks MB(r+1) and MB(r+2) are treated as skip blocks, inwhich portions of the bit stream Bs corresponding to the blocks MB(r)and MB(r+3) are shown.

Between a bit stream Bmb(r) corresponding to the block MB(r) and a bitstream Bmb(r+3) corresponding to the block MB(r+3), a skip identifierSf(Sk:2) indicating that there are two blocks regarded as skip blocksbetween these blocks is placed. Further, between a bit stream Bmb(r−1)corresponding to the block MB(r−1) and the bit stream Bmb(r)corresponding to the block MB(r), a skip identifier Sf(Sk:0) indicatingthat there is no block regarded as a skip block between these blocks isplaced.

The bit stream Bmb(r) corresponding to the block MB(r) is composed of aheader section Hmb and a data section Dmb, and the data section Dmbincludes coded image data corresponding to this block. Further, theheader section Hmb includes a mode flag Fm indicating a macroblock type,i.e., a coding mode in which this block is coded, reference pictureinformation Rp indicating a picture which is referred to in coding thisblock, and information Bmvf and Bmvb indicating motion vectors which areused in coding this block. This block MB(r) is coded by bidirectionalpredictive coding, and information Bmvf and Bmvb of the motion vectorsindicate the values of a forward motion vector and a backward motionvector which are used in the bidirectional predictive coding,respectively. Further, bit streams corresponding to other blocks, suchas a bit stream Bmb(r+3) corresponding to the block MB(r+3), have thesame structure as that of the bit stream Bmb(r) corresponding to theblock MB(r).

As described above, in the direct more, the amount of codes can bereduced by treating a block whose difference data becomes zero, as askip block, i.e., by skipping, in the bit stream, the informationcorresponding to this block together with the mode information.

Whether a block is skipped or not can be detected from the skipidentifier Sf that is placed just before the bit stream of each block.Further, whether a block is skipped or not can be known from blocknumber information or the like, that is described in the bit streamcorresponding to each block.

Furthermore, in the direct mode processing shown in FIG. 4(a) (firstexample), the direct mode processing shown in FIG. 4(b) (secondexample), and the direction mode processing shown in FIG. 5(a) (thirdexample), all of blocks whose difference data become zero are notnecessarily treated as skip blocks. That is, a target block is subjectedto bidirectional prediction using a picture that is positioned justprevious to the target picture as a forward reference picture, and amotion vector whose magnitude is zero, and only when the difference dataof the target block becomes zero, this target block may be treated as askip block.

By the way, selection of a coding mode for a target block is generallycarried out so as to minimize a coding error corresponding to apredetermined amount of bits. The coding mode determined by the modeselection unit 109 is output to the bit stream generation unit 104.Further, prediction data that is obtained from the reference pictureaccording to the coding mode determined in the mode selection unit 109is output to the difference calculation unit 102 and the addition unit106. However, when intra-picture coding is selected, no prediction datais outputted. Further, when the mode selection unit 109 selectsintra-picture coding, the switch 111 is controlled such that the inputterminal Ta is connected to the output terminal Tb1, and the switch 112is controlled such that the output terminal Td is connected to the inputterminal Tc1. When inter-picture predictive coding is selected, theswitch 111 is controlled such that the input terminal Ta is connected tothe output terminal Tb2, and the switch 112 is controlled such that theoutput terminal Td is connected to the input terminal Tc2.

Hereinafter, a description will be given of the operation of the movingpicture coding apparatus 10 in the case where the mode selection unit109 selects inter-picture predictive coding.

The difference calculation unit 102 receives the prediction data Pdoutputted from the mode selection unit 109. The difference calculationunit 102 calculates difference data between image data corresponding toa target block in the picture B11 and the prediction data, and outputsthe difference data as prediction error data PEd. The prediction errordata PEd is input to the prediction error coding unit 103. Theprediction error coding unit 103 subjects the inputted prediction errordata PEd to coding processes, such as frequency conversion andquantization, thereby generating coded data Ed. The coded data Edoutputted from the prediction error coding unit 103 is input to the bitstream generation unit 104 and the prediction error decoding unit 104.

The bit stream generation unit 104 subjects the inputted coded data Edto variable-length coding, and adds information such as a motion vectorand a coding mode to the coded data Ed to generate a bit stream Bs, andoutputs this bit stream Bs. When the coding mode is one performingforward reference, information (reference picture information) Rpindicating which one of the pictures P7, B9, and P10 is referred to whendetecting the forward motion vector is also added to the bit stream Bs.

Next, a description will be given of a method for managing the framememory, and a method for assigning reference picture information, incoding of the picture B11, with reference to FIG. 3.

When coding of the picture B11 is started, pictures P4, P7, P10, P13,and B9 are stored in the frame memory 117. The picture B11 is subjectedto bidirectional predictive coding, using the pictures P7, B9, and P10as candidate pictures for a forward reference, and the picture P13 as acandidate picture for a backward reference picture. The already-codedpicture B11 is stored in the memory area (#2) where the picture P4 hadbeen stored, because the picture P4 is not used as a reference picturein coding the pictures from the picture B11 onward.

In coding the picture B11, as a method for adding information indicatingwhich one of the pictures P7, B9, and P10 is referred to in detectingthe forward motion vector for the target block (reference pictureinformation), there is employed a method of successively assigningindices to the reference candidate pictures, starting from one that istimewise closest to the target picture (picture B11). The referencecandidate pictures are pictures which can be selected as a referencepicture in coding the target picture.

To be specific, the picture P10 is assigned a reference picture index[0], the picture B9 is assigned a reference picture index [1], and thepicture 7 is assigned a reference picture index [2].

Accordingly, when the picture P10 is referred to in coding the targetpicture, the reference picture index [0] is described in the bit streamcorresponding to the target block, as information indicating that acandidate picture just previous to the target picture is referred to.Likewise, when the picture B9 is referred to, the reference pictureindex [1] is described in the bit stream corresponding to the targetblock, as information indicating that a candidate picture two-picturesprevious to the target picture is referred to. Further, when the pictureP7 is referred to, the reference picture index [2] is described in thebit stream corresponding to the target block, as information indicatingthat a candidate picture three-pictures previous to the target pictureis referred to.

Assignment of codes to the reference picture indices [0], [1], and [2]is carried out such that a code of a shorter length is assigned to asmaller index.

Generally, a candidate picture that is timewise closer to a targetpicture is more likely to be used as a reference picture. Accordingly,by assigning the codes as described above, the total amount of codes,each indicating which one of plural candidate pictures is referred to indetecting the motion vector of the target block, can be reduced.

The prediction error decoding unit 105 subjects the inputted coded datacorresponding to the target block to decoding processes such as inversequantization and inverse frequency conversion to generate decodeddifference data PDd of the target block. The decoded difference data PDdis added to the prediction data Pd in the addition unit 106, and decodeddata Dd of the target picture which is obtained by the addition isstored in the frame memory 117.

The remaining blocks in the picture B11 are coded in like manner asdescribed above. When all of the blocks in the picture B11 have beenprocessed, coding of the picture B12 takes place.

Coding Process for Picture B12

Next, the coding process for the picture B12 will be described.

Since the picture B12 is a B picture, inter-picture predictive coding tobe performed for a target block in the picture B12 is bidirectionalinter-picture predictive coding in which two already-coded pictures thatare positioned timewise forward or backward the target picture arereferred to.

Hereinafter, a description will be given of a case where inter-picturepredictive coding using bidirectional reference is performed as a codingprocess for the picture B12. Accordingly, in this case, as candidatesfor a forward reference picture, two I or P pictures positioned close tothe target picture in order of display times or a B picture positionedclosest to the target picture in order of display times are/is used.Further, as a backward reference picture, an I or P picture positionedclosest to the target picture in order of display times is used.Accordingly, reference candidate pictures for the picture B12 are thepictures P7, P10, and B11 (forward pictures) and the picture P 13(backward picture).

In coding a B picture to be used as a reference picture when codinganother picture, the coding control unit 110 controls the respectiveswitches so that the switches 113, 114, and 115 are turned ON. Since thepicture B12 is to be used as a reference picture in coding anotherpicture, the coding control unit 110 controls the respective switches sothat the switches 113, 114, and 115 are turned ON. Accordingly, theimage data corresponding to the block in the picture B12, which is readfrom the frame memory 101, is input to the motion vector detection unit108, the mode selection unit 109, and the difference calculation unit102.

The motion vector detection unit 108 detects a forward motion vector anda backward motion vector corresponding to the target block in thepicture B12, using the pictures P7, P10, and B11 stored in the framememory 117 as forward reference candidate pictures, and the picture P13stored in the frame memory 117 as a backward reference picture.

The detected motion vectors are output to the mode selection unit 109.

The mode selection unit 109 determines a coding mode for the targetblock in the picture B12, using the motion vectors detected by themotion vector detection unit 108. For example, a coding mode for the Bpicture B12 is selected from among the intra-picture coding mode, theinter-picture predictive coding mode using a forward motion vector, theinter-picture predictive coding mode using a backward motion picture,the inter-picture predictive coding mode using bidirectional motionvectors, and the direct mode. When the coding mode is inter-picturepredictive coding using a forward motion vector, a most suitable pictureis selected as a reference picture from among the pictures P7, P10, andB11.

Hereinafter, a process of coding the blocks in the picture B12 by thedirect mode will be described.

First Example of Direct Mode Coding

FIG. 7(a) shows a case where a block (target block) BLa5 in the picture(target picture) B12 is coded in the direct mode. This direct modecoding utilizes a motion vector (base motion vector) MVc5 of a block(base block) BLb5 which is included in the picture (base picture) P13 asa reference picture positioned backward the picture B12 and is locatedin the same position as the target block BLa5. The motion vector MVc5 isa motion vector which is used when coding the block BLb5, and it isstored in the motion vector storage unit 116. This motion vector MVc5indicates an area CRc5 in the picture P10, which area corresponds to theblock BLb5. The block BLa5 is subjected to bidirectional predictivecoding, using motion vectors parallel to the motion vector MVc5, on thebasis of the pictures B11 and P13 as reference pictures for the blockBLa5. The motion vectors to be used in coding the block BLa5 are amotion vector MVe5 indicating an area CRd5 in the picture B11,corresponding to the block BLa5, and a motion vector MVe5 indicating anarea CRe5 in the picture P13, corresponding to the block BLa5. Themagnitudes MVF and MVB of the motion vectors MVd5 and MVe5 can beobtained by the above-mentioned formulae (1) and (2), respectively.

Second Example of Direct Mode Coding

Next, a second example of direct mode coding will be described.

FIG. 7(b) shows a case where a block (target block) BLa6 in the picture(target picture) B12 is coded in the direct mode. This direct modecoding utilizes a motion vector (base motion vector) MVc6 of a block(base block) BLb6 which is included in the picture (base picture) P13 asa reference picture positioned backward the picture B12 and is locatedin the same position as the target block BLa6. The motion vector MVc6 isa motion vector which is used when coding the block BLb6, and it isstored in the motion vector storage unit 116. This motion vector MVc6indicates an area CRc6 in the picture P7, which area corresponds to theblock BLb6. The block BLa6 is subjected to bidirectional predictivecoding, using motion vectors parallel to the motion vector MVc6, on thebasis of the pictures B11 and P13 as reference pictures. The motionvectors to be used in coding the block BLa6 are a motion vector MVg6indicating an area CRg6 in the picture B11, corresponding to the blockBLa6, and a motion vector MVh6 indicating an area CRh6 in the pictureP13, corresponding to the block BLa6. The magnitudes MVF and MVB of themotion vectors MVg6 and MVh6 can be obtained by the above-mentionedformulae (1) and (2), respectively.

As described above, in the direct mode, the motion vector MVf6 of theblock BLb6, which is included in the picture to be referred to as abackward reference picture when coding the target block BLa6 and islocated in relatively the same position as the target block, is scaled,thereby obtaining the forward motion vector MVg6 and the backward motionvector MVh6 corresponding to the target block. Therefore, when thedirect mode is selected, it is not necessary to send information of themotion vector of the target block. Furthermore, since the already-codedpicture which is positioned closest to the target picture in order ofdisplay times is used as a forward reference picture, predictionefficiency can be improved.

Third Example of Direct Mode Coding

Next, a third example of direct mode coding will be described.

FIG. 8(a) shows a third example of a process for coding a block (targetblock) BLa7 in the picture (target picture) B12 by the direct mode.

This direct mode coding utilizes a motion vector (base motion vector)MVc7 of a block (base block) BLb7 which is included in the picture (basepicture) P13 as a reference picture positioned backward the picture B12and is located in the same position as the target block BLa7. The motionvector MVc7 is a motion vector which is used when coding the block BLb7,and it is stored in the motion vector storage unit 116. This motionvector MVc7 indicates an area CRc7 in the picture P7, which areacorresponds to the block BLb7. The block BLa7 is subjected tobidirectional predictive coding, using motion vectors parallel to themotion vector MVc7, the same picture as that referred to in coding theblock BLb7 (i.e., the picture P7) as a forward reference picture), andthe picture P13 as a backward reference picture. The motion vectors tobe used in coding the block BLa7 are a motion vector MVd7 indicating anarea CRd7 in the picture P7, corresponding to the block BLa7, and amotion vector MVe7 indicating an area CRe7 in the picture P13,corresponding to the block BLa7.

The magnitudes MVF and MVB of the motion vectors MVd7 and MVe7 can beobtained by the above-mentioned formulae (2) and (3), respectively.

When the picture which is referred to when coding the block BLb7 hasalready been deleted from the frame memory 117, a forward referencepicture that is timewise closest to the target picture may be used as aforward reference picture in the direct mode. The direct mode coding inthis case is identical to that described as the first example of directmode coding.

As described above, in the direct mode coding shown in FIG. 8(a), themotion vector MVf7 of the block BLb7, which is included in the pictureto be used as a backward reference picture when coding the target blockand is located in relatively the same position as the target block, isscaled, thereby obtaining the forward motion vector MVd7 and thebackward motion vector MVe7 corresponding to the target block.Therefore, when the direct mode is selected, it is not necessary to sendinformation of the motion vector of the target block.

Fourth Example of Direct Mode Coding

Next, a fourth example of direct mode coding will be described.

FIG. 8(b) shows a fourth example of a process of coding a block (targetblock) BLa8 in the picture (target picture) B12 by the direct mode.

In this case, the target block BLa8 is subjected to bidirectionalpredictive coding with a motion vector being zero, on the basis of theclosest picture P10 that is selected as a forward reference picture, andthe picture P13 as a backward reference picture. That is, motion vectorsMVf8 and MVh8 to be used for coding the block BLa8 are a motion vectorindicating an area (block) CRf8 that is included in the picture B11 andis located in relatively the same position as the target block BLa8, anda motion vector indicating an area (block) CRh8 that is included in thepicture P13 and is located in relatively the same position as the targetblock BLa8, respectively.

As described above, in the direct mode coding shown in FIG. 8(b), themotion vector of the target block is forcefully set to zero. Therefore,when the direct mode is selected, it is not necessary to sendinformation of the motion vector of the target block, and scaling of themotion vector becomes unnecessary, resulting in a reduction incomplexity of signal processing. This method is applicable to, forexample, a case where a block which is included in the picture P13 as abackward reference picture of the picture B12 and is located in the sameposition as the block BLa8 is a block having no motion vector like anintra-frame-coded block. Accordingly, even when a block which isincluded in the backward reference picture and is located in the sameposition as the target block is coded without a motion vector, codingefficiency can be enhanced using the direct mode.

The above-mentioned direct mode processing for the picture B12 (first tofourth examples) is applicable not only when the interval of picturedisplay times is constant but also when the interval of picture displaytimes is variable, as in the case of the picture B11 shown in FIG. 6(a).

Furthermore, in direct mode coding for the picture B12, like the directmode coding for the picture B11, when the difference data correspondingto the target block becomes zero, the prediction error coding unit 103does not generate coded data corresponding to the target block, and thebit stream generation unit 104 does not output a bit streamcorresponding to the target block. Thus, a block whose difference databecomes zero is treated as a skip block, as in the case of the pictureB11 shown in FIGS. 6(b) and 6(c).

Furthermore, in the direct mode processing shown in FIG. 7(a) (firstexample), the direct mode processing shown in FIG. 7(b) (secondexample), and the direction mode processing shown in FIG. 8(a) (thirdexample), all of blocks whose difference data become zero are notnecessarily treated as skip blocks. That is, a target block is subjectedto bidirectional prediction using a picture that is positioned justprevious to the target picture as a forward reference picture, and amotion vector whose magnitude is zero, and only when the difference dataof the target block becomes zero, this target block may be treated as askip block.

When the coding mode for the target block in the picture B12 isdetermined by the mode selection unit 109, prediction data PEd for thetarget block is generated and outputted to the difference calculationunit 102 and the addition unit 106, as in the coding process for thetarget block in the picture B11. However, when intra-picture coding isselected, no prediction data is output from the mode selection unit 109.Further, the switches 111 and 112 are controlled in like manner asdescribed for coding of the picture B11, according to that eitherintra-picture coding or inter-picture coding is selected as a codingmode by the mode selection unit 109.

Hereinafter, a description will be given of the operation of the movingpicture coding apparatus 10 in the case where the mode selection unit109 selects inter-picture predictive coding when coding the picture P12.

In this case, the difference calculation unit 102, the prediction errorcoding unit 103, the bit stream generation unit 104, the predictionerror decoding unit 105, the addition unit 106, and the frame memory 117are operated in like manner as described for the case where the modeselection unit 109 selects inter-picture predictive coding for codingthe picture P11.

In this case, however, since the candidate pictures for a forwardreference picture are different from those for coding the picture P11,when the coding mode for the target block is one performing forwardreference, reference picture information to be added to the bit streamof the target block becomes information indicating which one of thepictures P7, P10, and B11 is referred to in detecting the forward motionvector.

Further, a frame memory management method and a reference pictureinformation assignment method which are to be employed in coding thepicture B12 are identical to those employed in coding the picture B11shown in FIG. 3.

As described above, according to the first embodiment of the invention,when coding a B picture (target picture), a B picture is used as acandidate picture for a forward reference picture as well as P pictures.Therefore, a forward picture positioned closest to the target B picturecan be used as a reference picture for the target B picture, wherebyprediction accuracy of motion compensation for the B picture can beincreased, resulting in an increase in coding efficiency.

In this first embodiment, no B picture is used as a reference picture incoding a P picture. Therefore, even when an error occurs in a pictureduring decoding, recovery from the error can be perfectly performed byresuming decoding from an I or P picture next to the picture where thedecoding error occurs. However, the other effects obtained by the firstembodiment are not changed even when a B picture is used as a referencepicture in coding a P picture.

Further, since two P pictures and one B picture are used as candidatepictures for a forward reference picture in coding a B picture, thenumber of candidate pictures for a forward reference picture for a Bpicture is not changed in comparison with the conventional case wherethree P pictures are used as candidate pictures for a forward referencepicture for a B picture. Therefore, it is possible to avoid an increasein the capacity of the frame memory for holding reference candidatepictures and an increase in processing amount for motion detection,which increases are caused by the inclusion of the B picture in thecandidate pictures for a forward reference picture for a B picture.

Further, in this first embodiment, information indicating that a Bpicture is subjected to inter-picture prediction coding with referenceto a forward B picture, and information indicating how many I or Ppictures and how many B pictures are used as candidate pictures forforward reference, are described as header information of a bit streamto be generated. Therefore, it is possible to know the capacity of aframe memory that is needed when decoding the bit stream generated inthe moving picture coding apparatus.

Furthermore, when information such as a motion vector, a coding mode,and the like is added to a bit stream, if the coding mode is oneperforming forward reference, reference picture information foridentifying reference pictures, which is assigned to candidate picturesto be referred to, is added to the bit stream, and further, referencepicture information assigned to a candidate picture that is timewiseclosest to the target picture is expressed with a code of a shorter codelength, according to a method of managing the frame memory for referencepictures. Therefore, the total amount of codes expressing the referencepicture information can be reduced. Further, in managing the framememory, since the frame memory is managed regardless of the picturetype, the capacity of the frame memory can be minimized.

Moreover, in this first embodiment, when the frame memory for referencepictures is managed with an area for P pictures and an area for Bpictures being separated from each other, management of the frame memoryis facilitated.

Further, when a block in a B picture is coded in the direct mode, apicture that is positioned closest to this B picture in order of displaytimes is used as a forward reference picture, whereby predictionefficiency in the direct mode for the B picture can be improved.

Furthermore, when a block in a B picture is to be coded in the directmode, a picture that is forward referred to in coding a backwardreference picture is used as a forward reference picture, wherebyprediction efficiency in the direct mode for the B picture can beimproved.

Furthermore, when a block in a B picture is to be coded in the directmode, bidirectional prediction with a motion vector being zero iscarried out on the basis of a forward reference picture and a backwardreference picture, whereby scaling of the motion vector in the directmode becomes unnecessary, resulting in a reduction in complexity ofinformation processing. In this case, even when a block which isincluded in the backward reference picture and is located in the sameposition as the target block is coded without a motion vector, codingefficiency can be enhanced using the direct mode.

Furthermore, when a block in a B picture is to be coded in the directmode, if a prediction error with respect to the target block becomeszero, information relating to the target block is not described in thebit stream, whereby the amount of codes can be reduced.

In this first embodiment, motion compensation is performed in units ofimage spaces (macroblocks) each comprising 16 pixels in the horizontaldirection×16 pixels in the vertical direction, and coding of aprediction error image is performed in units of image spaces (subblocks)each comprising 8 pixels in the horizontal direction×8 pixels in thevertical direction. However, the number of pixels in each macroblock(subblock) in motion compensation (coding of a prediction error image)may be different from that described for the first embodiment.

Further, while in this first embodiment the number of continuous Bpictures is two, the number of continuous B pictures may be three ormore.

For example, the number of B pictures placed between an I picture and aP picture or between two P pictures may be three or four.

Further, in this first embodiment, a coding mode for a P picture isselected from among intra-picture coding, inter-picture predictivecoding using a motion vector, and inter-picture predictive coding usingno motion vector, while a coding mode for a B picture is selected fromamong the intra-picture coding, the inter-picture predictive codingusing a forward motion vector, the inter-picture predictive coding usinga backward motion vector, the inter-picture predictive coding usingbidirectional motion vectors, and the direct mode. However, the codingmode for a P picture or a B picture may be other than those mentionedabove.

For example, when the direct mode is not used as a coding mode for a Bpicture, the motion vector storage unit 116 in the moving picture codingapparatus 10 is dispensed with.

Further, while in this first embodiment the picture B11 or B12 as a Bpicture becomes a candidate picture for a reference picture in codinganother picture, it is not necessary to store a B picture which is notto be used as a reference picture in coding another picture, in thereference picture memory 117. In this case, the coding control unit 110turns off the switches 114 and 115.

Further, while in this first embodiment three pictures are used ascandidate pictures for forward reference in coding a P picture, thepresent invention is not restricted thereto. For example, two picturesor four or more pictures may be used as candidate pictures for forwardreference in coding a P picture.

While in this first embodiment two P pictures and one B picture are usedas candidate pictures for forward reference in coding a B picture,candidate pictures for forward reference in coding a B picture are notrestricted to those mentioned above.

For example, in coding a B picture, candidate pictures for forwardreference may be one P picture and two B pictures, or two P pictures andtwo B pictures, or three pictures closest to the target pictureregardless of the picture type. Further, not a B picture closest to thetarget picture on the display time axis but a B picture apart from thetarget picture on the display time axis may be used as a referencecandidate picture.

Further, in a case where, in coding a block in a B picture, one backwardpicture is referred to and only one picture closest to the targetpicture is used as a candidate picture for forward reference, it is notnecessary to describe information indicating which picture is referredto in coding the target block (reference picture information) in the bitstream.

Further, in this first embodiment, when coding a B picture, a B picturewhich is positioned forward a P picture that is positioned forward andclosest to the target picture is referred to. However, in coding a Bpicture, a B picture which is positioned forward an I or P picture thatis forward and closest to the target picture is not necessarily referredto. In this case, when decoding a generated bit stream, even if an erroroccurs during the decoding, recovery from the error can be perfectlycarried out by resuming decoding from an I or P picture next to thepicture where the error occurs.

For example, FIGS. 9(a) and 9(b) are diagrams illustrating a case where,when coding a B picture, a B picture which is positioned forward a Ppicture that is positioned forward and closest to the target picture, isreferred to.

FIG. 9(a) illustrates a picture arrangement, and relationships between Bpictures and reference pictures. To be specific, in FIG. 9(a), two Bpictures are positioned between adjacent P pictures, and one P pictureand two B pictures are used as candidate pictures for a forwardreference picture for a B picture (i.e., a picture to be referred towhen coding the target B picture).

FIG. 9(b) illustrates another picture arrangement, and relationshipsbetween B pictures and reference pictures. To be specific, in FIG. 9(b),four B pictures are positioned between adjacent P pictures, and twopictures which are timewise closest to the target picture, regardless ofthe picture type, are used as candidate pictures for a forward referencepicture for a B picture.

Further, FIGS. 10(a) and 10(b) are diagrams illustrating a case where,when coding a B picture, a B picture which is positioned forward an I orP picture that is positioned forward and closest to the target picture,is not referred to.

To be specific, in FIG. 10(a), two B pictures are positioned betweenadjacent P pictures, one P picture and one B picture are used ascandidate pictures for a forward reference picture for a B picture, anda B picture which is positioned forward a P picture that is positionedforward and closest to the target picture is not used as a candidatepicture for the forward reference picture.

In FIG. 10(b), four B pictures are positioned between adjacent Ppictures, one P picture and one B picture are used as candidate picturesfor a forward reference picture for a B picture, and a B picture whichis positioned forward a P picture that is positioned forward and closestto the target picture is not used as a candidate picture for the forwardreference picture.

Further, in this first embodiment, three pictures are used as referencecandidate pictures for a P picture, and two P pictures and one B pictureare used as candidate pictures for forward reference for a B picture,i.e., the number of pictures which can be referred to when coding a Ppicture is equal to the number of pictures which can be forward referredto when coding a B picture. However, the number of pictures which can beforward referred to when coding a B picture may be less than the numberof pictures which can be referred to when coding a P picture.

Furthermore, while in this first embodiment four methods are describedas examples of direct mode coding, one of these four methods or some ofthese four methods may be employed in the direct mode. However, whenemploying plural methods, it is desirable to describe informationindicating the employed direct modes (DM mode information) in the bitstream.

For example, when one method is used over the whole sequence, DM modeinformation is described in the header of the whole sequence. When onemethod is selected for each picture, DM mode information is described inthe header of the picture. When one method is selected for each block,DM mode information is described in the header of the block.

Although a picture or a block is described as a unit for which one ofthe direct mode coding methods is selected, it may be a GOP (Group ofPictures) comprising plural pictures, a GOB (Group of Blocks) comprisingplural blocks, or a slice which is obtained by dividing a picture.

Further, while in this first embodiment a frame memory managing methodis described with reference to FIG. 3, applicable frame memory managingmethods are not restricted to that shown in FIG. 3.

Hereinafter, other frame memory managing methods will be described.

Initially, a first example of a frame memory managing method in whichall pictures used as reference pictures are separated into P picturesand B pictures to be managed, will be described with reference to FIG.11.

In this case, the frame memory 117 has memory areas for six pictures intotal, i.e., P picture memory areas (#1)˜(#4) and B picture memory areas(#1) and (#2). A storage for each picture is not restricted to an areain the frame memory, and it may be one memory.

When coding of the picture P13 is started, pictures P1, P4, P7, and P10are stored in the P picture memory areas (#1)˜(#4) in the frame memory117, respectively, and pictures B8 and B9 are stored in the B picturememory areas (#1) and (#2), respectively. The picture P13 is coded usingthe pictures P4, P7, and P10 as candidate pictures for a referencepicture, and the coded picture P13 is stored in the area (#1) where thepicture P1 had been stored, because the picture P1 is not used as areference picture when coding the picture P13 and the subsequentpictures.

In this case, a method for assigning reference picture information tothe pictures P4, P7, and P10 as candidate pictures is identical to themethod shown in FIG. 3, that is, a smaller reference picture index isassigned to a candidate picture that is timewise closer to the targetpicture.

To be specific, a reference picture index [0] is assigned to a forwardcandidate picture that is closest to the target picture, a referencepicture index [1] is assigned to a candidate picture that issecond-close to the target picture, and a reference picture index [2] isassigned to a candidate picture that is most distant from the targetpicture.

In FIG. 11, pictures to be used as backward reference pictures areassigned codes [b] as reference picture information, and pictures whichare not used as reference pictures when coding the target picture andthe subsequent pictures are assigned codes [n].

Next, a second example of a frame memory managing method in which allpictures used as reference pictures are separated into P pictures and Bpictures to be managed, will be described with reference to FIG. 12.

Since memory management in this second example is identical to that inthe first example shown in FIG. 11, repeated description is notnecessary.

In this second example, as a method for assigning reference pictureindices, assignment of indices to the pictures stored in the P picturememory areas is carried out with priority. However, in coding thepicture P 13, since no B pictures are used as reference pictures, noindices are assigned to the B pictures. Accordingly, a reference pictureindex [0] is assigned to the picture P10, a reference picture index [1]is assigned to the picture P7, and a reference picture index [2] isassigned to the picture P4.

Next, a third example of a frame memory managing method in which allpictures used as reference pictures are separated into P pictures and Bpictures to be managed, will be described with reference to FIG. 13.

Since memory management in this third example is identical to that inthe first example shown in FIG. 11, repeated description is notnecessary.

In this third example, as a method for assigning reference pictureindices, assignment of indices to the pictures stored in the B picturememory areas is carried out with priority. However, in coding thepicture P 13, since no B pictures are used as reference pictures, noindices are assigned to the B pictures. Accordingly, a reference pictureindex [0] is assigned to the picture P10, a reference picture index [1]is assigned to the picture P7, and a reference picture index [2] isassigned to the picture P4.

Next, a fourth example of a frame memory managing method in which allpictures used as reference pictures are separated into P pictures and Bpictures to be managed, will be described with reference to FIG. 14.

Since memory management in this fourth example is identical to that ofthe first example shown in FIG. 11, repeated description is notnecessary.

In this third example, as a method for assigning reference pictureindices, either the pictures stored in the P picture memory area or thepictures stored in the B picture memory area are selected for eachtarget picture to be coded, and reference picture indices are given tothe selected pictures with priority.

To be specific, according to the type of a reference picture that istimewise closest to the target picture, it is determined that either thepicture stored in the P picture memory area or the picture stored in theB picture memory area should be given priority in assigning referencepicture indices.

In coding the picture P13, since no B picture is used as a referencepicture, reference picture indices are assigned to the pictures storedin the P picture memory area with priority. Accordingly, a referencepicture index [0] is assigned to the picture P10, a reference pictureindex [1] is assigned to the picture P7, and a reference picture index[2] is assigned to the picture P4. In this case, information indicatingthat the reference picture indices are given to the pictures stored inthe P picture memory areas with priority, is described in the header ofeach picture.

In the reference picture index assigning methods shown in FIGS. 3 and 11to 14, the smaller the reference picture index is, the shorter thelength of a code indicating the reference picture index is. Since,generally, a picture that is timewise closer to the target picture ismore likely to be used as a reference picture, the total amount of codesexpressing the reference picture indices can be reduced by determiningthe lengths of the codes expressing the reference picture indices, asmentioned above.

While the five methods shown in FIGS. 3 and 11˜14 are described relatingto frame memory management and reference picture index assignment, oneof these methods may be previously selected for use. Further, some ofthese methods may be used by switching them. In this case, however, itis desirable to describe information about the methods being used, asheader information or the like.

Further, when information indicating that each P picture is subjected tointer-picture predictive coding using three reference candidate picturesis described as header information, it is possible to know the capacityof a frame memory that is needed in decoding the bit stream Bs generatedin the moving picture coding apparatus 10 according to the firstembodiment. These header information may be described in the header ofthe whole sequence, the header of each GOP (Group of Pictures)comprising plural pictures, or the header of each picture.

Subsequently, as a frame memory managing method and a reference pictureinformation assigning method to be used in coding the picture B11,methods other than that shown in FIG. 3 (i.e., methods of separating thereference candidate pictures into P pictures and B pictures formanagement) will be described.

Initially, a description will be given of a first example of a methodfor separating the reference candidate pictures into P pictures and Bpictures to be managed, with reference to FIG. 11.

When coding of the picture B11 is started, in the frame memory 117,pictures P4, P7, P10, and P13 are stored in the P picture memory areaswhile pictures B8 and B9 are stored in the B picture memory areas. Thepicture P11 is coded using the pictures P7, B9, and P10 as candidatepictures for forward reference and the picture P13 as a candidatepicture for backward reference, and then the coded picture P11 is storedin the area where the picture P8 had been stored, because the picture P8is not used as a reference picture in coding the picture P11 and thesubsequent pictures.

In this case, as a method for assigning reference picture information toeach picture (i.e., information indicating which one of the referencecandidate pictures P7, B9, and P10 is used as a reference picture indetecting the forward motion vector), a method for assigning referencepicture indices to the reference candidate pictures, starting from onethat is timewise closest to the target picture, is used as describedwith respect to FIG. 3.

That is, a reference picture index [0] is assigned to a candidatepicture (picture P10) that is just previous to the target picture(picture B11), a reference picture index [1] is assigned to a candidatepicture (picture B9) that is two-pictures previous to the targetpicture, and a reference picture index [3] is assigned to a candidatepicture (picture 7) that is three-pictures previous to the targetpicture.

Next, a second example of a frame memory managing method in whichreference candidate pictures are separated into P pictures and Bpictures to be managed in coding the picture B11, will be described withreference to FIG. 12.

Since memory management in this second example is identical to that inthe first example shown in FIG. 11, repeated description is notnecessary.

In this second example, as a method for assigning reference pictureindices, assignment of indices to the pictures stored in the P picturememory areas is carried out with priority. Accordingly, a referencepicture index [0] is assigned to the picture P10, a reference pictureindex [1] is assigned to the picture P7, and a reference picture index[2] is assigned to the picture B9.

Next, a third example of a frame memory managing method in whichreference candidate pictures are separated into P pictures and Bpictures to be managed in coding the picture B11, will be described withreference to FIG. 13.

Since memory management in this third example is identical to that inthe first example shown in FIG. 11, repeated description is notnecessary.

In this third example, as a method for assigning reference pictureindices, assignment of indices to the pictures stored in the B picturememory areas is carried out with priority. Accordingly, a referencepicture index [0] is assigned to the picture B9, a reference pictureindex [1] is assigned to the picture P10, and a reference picture index[2] is assigned to the picture P7.

Next, a fourth example of a frame memory managing method in whichreference candidate pictures are separated into P pictures and Bpictures to be managed in coding the picture B11, will be described withreference to FIG. 14.

Since memory management in this fourth example is identical to that inthe first example shown in FIG. 11, repeated description is notnecessary.

In this fourth example, as a method for assigning reference pictureindices, either the pictures stored in the P picture memory areas or thepictures stored in the B picture memory areas are selected for eachtarget picture to be coded, and reference picture indices are given tothe selected pictures with priority.

To be specific, it is determined which one of the P picture memory andthe B picture memory should be assigned reference picture indices,according to the type of the reference candidate picture that istimewise closest to the target picture to be coded.

In coding the picture B11, since the forward reference picture that istimewise closest to the target picture is the picture P10, referencepicture indices are assigned to the pictures stored in the P picturememory area with priority.

Accordingly, a reference picture index [0] is assigned to the pictureP10, a reference picture index [1] is assigned to the picture P7, and areference picture index [2] is assigned to the picture B9. In this case,information indicating that the reference picture indices are given tothe pictures stored in the P picture memory areas with priority, isdescribed in the header of each picture.

In the methods of assigning reference picture indices when coding thepicture B11 (the five methods shown in FIGS. 3 and 11 to 14), as in thecase of coding the picture P13, the smaller the reference picture indexis, the shorter the length of a code indicating the reference pictureindex is.

Further, in coding the B picture B11, as in the case of coding the Ppicture P13, one of the five methods may previously be selected for use.Further, some of these methods may be used by switching them. However,in the case where plural methods are used by switching them, it isdesirable that information about the methods being used should bedescribed as header information or the like.

Further, by describing, as header information, information indicatingthat a B picture is subjected to inter-picture predictive coding using aforward B picture as a reference candidate picture, and informationindicating that candidate pictures for forward reference, which are usedin coding the B picture, are two I or P pictures and one B picture, itis possible to know the storage capacity of a frame memory that isneeded in decoding the bit stream generated in the moving picture codingapparatus 10 according to the first embodiment. These header informationmay be described in the header of the whole sequence, the header of eachGOP (Group of Pictures) comprising plural pictures, or the header ofeach picture.

Finally, as a frame memory managing method and a reference pictureinformation assigning method to be employed in coding the picture B12,methods other than that shown in FIG. 3 (i.e., methods of separatingreference candidate pictures into P pictures and B pictures to bemanaged) will be described.

Since the first to third examples shown in FIGS. 11 to 13 are identicalto those in the case of coding the picture B11, repeated description isnot necessary.

So, only a fourth example of managing reference candidate pictures beingseparated into P pictures and B pictures will be described for thepicture B12, with reference to FIG. 14.

Since memory management in this fourth example is identical to that inthe first example in which reference candidate pictures are separatedinto P pictures and B pictures to be managed in coding the picture B11,repeated description is not necessary.

In this fourth example, as for a method of assigning, to each picture,information indicating which of the reference candidate pictures P7,P10, and B11 is referred to in detecting the forward motion vector, amethod of determining, for each picture to be coded, either thecandidate pictures stored in the P picture memory areas or the candidatepictures stored in the B picture memory areas should be given priorityis used.

To be specific, in coding the picture B12, which of the candidatepicture in the P picture memory area and that in the B picture memoryarea should be assigned a reference picture index with priority, isdetermined according to the type of the reference picture that istimewise closest to the target picture.

In coding the picture B12, since the forward reference candidate picturethat is timewise closest to the target picture (picture B12) is thepicture B11, the pictures stored in the B picture memory areas areassigned indices with priority.

Accordingly, a reference picture index [0] is assigned to the pictureB11, a reference picture index [1] is assigned to the picture P10, and areference picture index [2] is assigned to the picture P7. In this case,information indicating that assignment of reference picture indices tothe pictures in the B picture memory areas takes priority is describedin the header of each picture.

Further, as in the case of coding the picture B11, there are described,as header information, that the B picture is subjected to inter-picturepredictive coding using also the forward B picture as a referencecandidate picture, and that the forward reference candidate picturesused in coding the B picture are two I or P pictures and one B picture.

Furthermore, in this first embodiment, the five examples of frame memorymanaging methods (FIGS. 3, 11˜14) are described with respect to the casewhere there are three reference candidate pictures for a P picture, andthere are two P pictures and one B picture as forward referencecandidate pictures for a B picture. However, each of the five examplesof frame memory management methods may be applied to cases where thenumber of reference candidate pictures is different from those mentionedfor the first embodiment. When the number of reference candidatepictures differs from those of the first embodiment, the capacity of theframe memory differs from that of the first embodiment.

Further, in this first embodiment, in the methods of managing the framememory in which reference candidate pictures are separated into Ppictures and B pictures (four examples shown in FIGS. 11˜14), P picturesare stored in the P picture memory areas while B pictures are stored inthe B picture memory areas. However, a short-term picture memory and along-term picture memory which are defined in H.263++ may be used asmemory areas where pictures are stored. For example, the short-termpicture memory and the long-term picture memory may be used as a Ppicture memory area and a B picture memory area, respectively.

Embodiment 2

Hereinafter, a second embodiment of the present invention will bedescribed.

FIG. 15 is a block diagram for explaining a moving picture decodingapparatus 20 according to a second embodiment of the present invention.

The moving picture decoding apparatus 20 decodes the bit stream Bsoutputted from the moving picture coding apparatus 10 according to thefirst embodiment.

To be specific, the moving picture decoding apparatus 20 includes a bitstream analysis unit 201 for analyzing the bit stream Bs to extractvarious kinds of data; a prediction error decoding unit 202 for decodingcoded data Ed outputted from the bit stream analysis unit 201 to outputprediction error data PDd; and a mode decoding unit 223 for outputting aswitch control signal Cs on the basis of mode information (coding mode)Ms relating to mode selection, which is extracted by the bit streamanalysis unit 201.

The moving picture decoding apparatus 20 further includes a referencepicture memory 207 for holding decoded image data DId, and outputtingthe stored image data as reference data Rd or output image data Od; amotion compensation decoding unit 205 for generating prediction data Pdon the basis of the data (reference image data) Rd that is read from thereference picture memory 207, information of a motion vector MV that isextracted by the bit stream analysis unit 201, and the coding mode Msthat is output from the mode decoding unit 223; and an addition unit 208for adding the prediction data Pd to the output data PDd from theprediction error decoding unit 202 to generate decoded data Ad.

The moving picture decoding apparatus 20 further includes a memorycontrol unit 204 for controlling the reference picture memory 207 with amemory control signal Cm on the basis of header information Ih that isextracted by the bit stream analysis unit 201; a selection switch 209placed between the prediction error decoding unit 202 and the additionunit 208; and a selection switch 210 placed between the addition unit208 and the reference picture memory 207.

The selection switch 201 has one input terminal Te and two outputterminals Tf1 and Tf2, and the input terminal Te is connected to one ofthe output terminals Tf1 and Tf2, according to the switch control signalCs. The selection switch 210 has two input terminals Tg1 and Tg2 and anoutput terminal Th, and the output terminal Th is connected to one ofthe input terminals Tg1 and Tg2, according to the switch control signalCs. Further, in the selection switch 209, the output data PDd from theprediction error decoding unit 202 is applied to the input terminal Te,and the output data PDd from the prediction error decoding unit 202 isoutput from one output terminal Tf1 to the input terminal Tg1 of theselection switch 210 while the output data PDd is output from the otheroutput terminal Tf2 to the addition unit 208. In the selection switch210, the output data PDd from the prediction error decoding unit 202 isapplied to one input terminal Tg1 while the output data Ad from theaddition unit 208 is input to the other input terminal Tg1, and eitherthe output data PDd or the output data Ad is output from the outputterminal Th to the reference picture memory 207 as decoded image dataDId.

Further, the moving picture decoding apparatus 20 includes a motionvector storage unit 226 for holding the motion vector MV from the motioncompensation decoding unit 205, and outputting the stored motion vectorMV to the motion compensation decoding unit 205.

Next, the operation will be described.

In the following description, a picture which is positioned forward orbackward a target picture to be decoded on a display time axis isreferred to as a picture which is timewise forward or backward thetarget picture, or simply as a forward picture or a backward picture.

The bit stream Bs generated in the moving picture coding apparatus 10 ofthe first embodiment is input to the moving picture decoding apparatus20 shown in FIG. 15. In this second embodiment, a bit stream of a Ppicture is obtained by performing inter-picture predictive coding withreference to one picture selected from among three candidate pictures (Ior P pictures) which are positioned close to and timewise forward orbackward the P picture. Further, a bit stream of a B picture is obtainedby performing inter-picture predictive coding with reference to twopictures selected from among four candidate pictures positioned forwardor backward the B picture (i.e., forward two I or P pictures that aretimewise closest to the target picture, one B picture that is timewiseclosest to the target picture, and an I or P picture that is positionedtimewise backward the target picture). The four candidate pictures forthe target B picture include another B picture that is positionedtimewise forward the target B picture.

Further, which candidate pictures are referred to when coding the targetP picture or B picture may be described as header information of the bitstream. Accordingly, it is possible to know which pictures are referredto when coding the target picture, by extracting the header informationin the bit stream analysis unit 201. This header information Ih is alsooutput to the memory control unit 204.

In this case, coded data corresponding to pictures in the bit stream arearranged in coding order as shown in FIG. 16(a).

To be specific, the coded data of the pictures in the bit stream Bs arearranged in the other of P4, B2, B3, P7, B5, B6, P10, B8, B9, P13, B11,B12, P16, B14, and B15. In other words, in this picture arrangement, therespective pictures are successively arranged from one having earlierdecoding time on a decoding time axis Y that indicates decoding timesTdec of the pictures (arrangement in decoding order).

FIG. 16(b) shows an arrangement of pictures in which the picturesarranged in decoding order are rearranged in display order. That is, inFIG. 16(b), the pictures B2, B3, P4, B5, B6, P7, B8, B9, P10, B11, B12,P13, B14, B15, and p16 are successively arranged from one having earlierdisplay time on a display time axis X that indicates display times Tdisof the respective pictures (arrangement in display order).

Hereinafter, decoding processes for the pictures P13, B11, and B12 willbe described in this order.

Decoding Process for Picture P13

The Bit Stream of the Picture P13 is Input to the Bit Stream analysisunit 201. The bit stream analysis unit 201 extracts various kinds ofdata from the inputted bit stream. The respective data are as follows:information for performing mode selection, i.e., information indicatinga coding mode Ms (hereinafter referred to simply as a coding mode);information indicating a motion vector MV (hereinafter referred tosimply as a motion vector), header information, coded data (imageinformation), and the like. The extracted coding mode Ms is output tothe mode decoding unit 203. Further, the extracted motion vector MV isoutput to the motion compensation decoding unit 205. Furthermore, theprediction error coded data Ed extracted by the bit stream analysis unit201 is output to the prediction error decoding unit 202.

The mode decoding unit 203 controls the switches 209 and 210 on thebasis of the coding mode Ms extracted from the bit stream. When thecoding mode indicates inter-picture coding, the switch 209 is controlledsuch that the input terminal Te is connected to the output terminal Tf1,and the switch 210 is controlled such that the output terminal Th isconnected to the input terminal Tg1. Further, when the coding modeindicates inter-picture prediction coding, the switch 209 is controlledsuch that the input terminal Te is connected to the output terminal Tf2,and the switch 210 is controlled such that the output terminal Th isconnected to the input terminal Tg2. Further, the mode decoding unit 203outputs the coding mode Ms to the motion compensation decoding unit 205.

Hereinafter, a description will be given of the case where the codingmode is inter-picture predictive coding.

The prediction error decoding unit 202 decodes the inputted coded dataEd to generate prediction error data PDd. The generated prediction errordata PDd is output to the switch 209. In this case, since the inputterminal Te of the switch 209 is connected to the output terminal Tf2,the prediction error data PDd is output to the addition unit 208.

The motion compensation decoding unit 205 performs motion compensationon the basis of the motion vector MV and the reference picture index Rpwhich are extracted by the analysis unit 201, and obtains a motioncompensation image from the reference picture memory 207. This motioncompensation image is an image in an area in the reference picture,which area corresponds to a target block to be decoded.

The picture P13 has been coded using the pictures P4, P7, and P10 ascandidate pictures for forward reference. When decoding the picture P13,these candidate pictures have already been decoded and are stored in thereference picture memory 207.

So, the motion compensation decoding unit 205 determines which one ofthe pictures P4, P7, and P10 is used as a reference picture when codingthe target block of the picture P13. Then, the motion compensationdecoding unit 205 obtains an image in an area in the reference picture,which area corresponds to the target block, as a motion compensationimage from the reference picture memory 207 on the basis of the motionvector.

Hereinafter, a description will be given of how the pictures stored inthe reference picture memory 207 change with time, and a method fordetermining a reference picture, with reference to FIG. 3.

The reference picture memory 207 is controlled by the memory controlunit 204 on the basis of information indicating what kind of referencehas been carried out to obtain P pictures and B pictures (referencepicture information), which information is extracted from the headerinformation of the bit stream.

As shown in FIG. 3, the reference picture memory 207 has memory areas(#1)˜(#5) for five pictures. When decoding of the picture P13 isstarted, pictures B8, P4, P7, P10, and B9 are stored in the referencepicture memory 207. The picture P13 is decoded using the pictures P4,P7, and P10 as candidate pictures for a reference picture. The decodedpicture P13 is stored in the memory area where the picture P8 had beenstored. The reason is as follows. While the pictures P4, P7, and P10 areused as candidate pictures for a reference picture when decoding thepicture P13 and the following pictures, the picture B8 is not used as areference picture when decoding these pictures.

In FIG. 3, each circled picture is a picture (target picture) which isfinally stored in the reference picture memory 207 when decoding of thetarget picture has completed.

In this case, which picture has been referred to in detecting the motionvector of the target block in the picture P13 can be determined from thereference picture information that is added to the motion vector.

Concretely, the reference picture information is reference pictureindices, and the reference picture indices are assigned to the referencecandidate pictures for the picture P13. This assignment of the referencepicture indices to the reference candidate pictures is carried out suchthat a smaller index is assigned to a reference candidate picture thatis timewise closer to the target picture (picture P13).

To be specific, when the picture P10 has been referred to in coding thetarget block of the picture P13, information indicating that thecandidate picture (picture P10) just previous to the target picture hasbeen used as a reference picture (e.g., reference picture index [0]) isdescribed in the bit stream of the target block. Further, when thepicture P7 has been referred to in coding the target block, informationindicating that the candidate picture which is two-pictures previous tothe target picture has been used as a reference picture (e.g., referencepicture index [1]) is described in the bit stream of the target block.Furthermore, when the picture P4 has been referred to in coding thetarget block of the picture P13, information indicating that thecandidate picture which is three-pictures previous to the target picturehas been used as a reference picture (e.g., reference picture index [2])is described in the bit stream of the target block.

It is possible to know which one of the candidate pictures has been usedas a reference picture in coding the target block, by the referencepicture index.

In this way, the motion compensation decoding unit 205 obtains themotion compensation image, i.e., the image in the area in the referencepicture corresponding to the target block, from the reference picturememory 207, according to the motion vector and the reference pictureinformation.

The motion compensation image thus generated is output to the additionunit 208.

Further, when performing decoding of a P picture, the motioncompensation decoding unit 205 outputs the motion vector MV and thereference picture information Rp to the motion vector storage unit 226.

The addition unit 208 adds the prediction error data PDd and the data(prediction data) Pd of the motion compensation image to generatedecoded data Ad. The decoded data Ad so generated is output as decodedimage data DId, through the switch 210 to the reference picture memory207.

In this way, the blocks in the picture P13 are successively decoded.When all of the blocks in the picture P13 have been decoded, decoding ofthe picture B11 takes place.

Decoding Process for Picture B11

Since the bit stream analysis unit 201, the mode decoding unit 203, andthe prediction error decoding unit 202 operate in the same way asdescribed for decoding of the picture P13, repeated description is notnecessary.

The motion compensation decoding unit 205 generates motion compensationdata from the inputted information such as the motion vector. The bitstream analysis unit 201 outputs the motion vector and the referencepicture index to the motion compensation decoding unit 205. The pictureP11 is obtained by predictive coding using the pictures P7, B9 and P10as candidate pictures for forward reference, and the picture P13 as acandidate picture for backward reference. At decoding the targetpicture, these reference candidate pictures have already been decoded,and are stored in the reference picture memory 207.

Hereinafter, a description will be given of how the pictures stored inthe reference picture memory 207 change with time, and a method fordetermining a reference picture, with reference to FIG. 3.

The reference picture memory 207 is controlled by the memory controlunit 204 on the basis of information Ih indicating what kind ofreference has been carried out in coding P pictures and B pictures,which information is extracted from the header information of the bitstream.

When decoding of the picture P11 is started, pictures P13, P4, P7, P10,and B9 are stored in the reference picture memory 207 as shown in FIG.3. The picture B11 is decoded using the pictures P7, B9, and P10 ascandidate pictures for forward reference, and the picture P13 as abackward reference picture. The decoded picture B11 is stored in thememory area where the picture P4 had been stored, because the picture P4is not used as a candidate for a reference picture when decoding thepicture B11 and the following pictures.

In this case, which candidate picture has been referred to in detectingthe forward motion vector can be determined from the reference pictureinformation added to the motion vector.

To be specific, when the picture P10 has been referred to in coding thetarget block of the picture B11, information indicating that thecandidate picture (picture P10) just previous to the target picture hasbeen used as a reference picture (e.g., reference picture index [0]) isdescribed in the bit stream of the target block. Further, when thepicture B9 has been referred to in coding the target block, informationindicating that the candidate picture which is two-pictures previous tothe target picture has been used as a reference picture (e.g., referencepicture index [1]) is described in the bit stream of the target block.Furthermore, when the picture P7 has been referred to in coding thetarget block of the picture P13, information indicating that thecandidate picture which is three-pictures previous to the target picturehas been used as a reference picture (e.g., reference picture index [2])is described in the bit stream of the target block.

Accordingly, it is possible to know which one of the candidate pictureshas been used as a reference picture in coding the target block, fromthe reference picture index.

When the selected mode is bidirectional predictive coding, the motioncompensation decoding unit 205 determines which one of the pictures P7,B9 and P10 has been used for forward reference, from the referencepicture index. Then, the motion compensation decoding unit 205 obtains aforward motion compensation image from the reference picture memory 207on the basis of the forward motion vector, and further, it obtains abackward motion compensation image from the reference picture memory 207on the basis of the backward motion vector.

Then, the motion compensation decoding unit 205 performs addition andaveraging of the forward motion compensation image and the backwardmotion compensation image to generated a motion compensation image.

Next, a process of generating a motion compensation image using forwardand backward motion vectors will be described.

(Bidirectional Prediction Mode)

FIG. 17 illustrates a case where the target picture to be decoded is thepicture B11, and bidirectional predictive decoding is performed on ablock (target block) BLa01 to be decoded, in the picture B11.

Initially, a description will be given of a case where the forwardreference picture is the picture P10, and the backward reference pictureis the picture P13.

In this case, the forward motion vector is a motion vector MVe01indicating an area CRe01 in the picture P10, which area corresponds tothe block BLa01. The backward motion vector is a motion vector MVg01indicating an area CRg01 in the picture P13, which area corresponds tothe block BLa01.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRe01 in the picture P10 as a forward reference image, andan image in the area CRg01 in the picture P13 as a backward referenceimage, from the reference picture memory 207, and performs addition andaveraging of image data on the images in the both areas CRe01 and CRg01to obtain a motion compensation image corresponding to the target blockBLa01.

Next, a description will be given of a case where the forward referencepicture is the picture B9, and the backward reference picture is thepicture P13.

In this case, the forward motion vector is a motion vector MVf01indicating an area CRf01 in the picture B9, which area corresponds tothe block BLa01. The backward motion vector is a motion vector MVg01indicating an area CRg01 in the picture P13, which area corresponds tothe block BLa01.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRf01 in the picture B9 as a forward reference image, and animage in the area CRg01 in the picture P13 as a backward referenceimage, from the reference picture memory 207, and performs addition andaveraging of image data for the images in the both areas CRf01 and CRg01to obtain a motion compensation image corresponding to the target blockBLa01.

(Direct Mode)

Further, when the coding mode is the direct mode, the motioncompensation decoding unit 205 obtains a motion vector (base motionvector) of a block that is included in the backward reference pictureP13 for the target picture B11 and is placed relatively in the sameposition as the target block, which motion vector is stored in themotion vector storage unit 226. The motion compensation decoding unit205 obtains a forward reference image and a backward reference imagefrom the reference picture memory 207 by using the base motion vector.Then, the motion compensation decoding unit 205 performs addition andaveraging of image data for the forward reference image and the backwardreference image, thereby generating a motion compensation imagecorresponding to the target block. In the following description, a blockin a picture, whose relative position with respect to a picture is equalto that of a specific block in another picture is simply referred to asa block which is located in the same position as a specific block in apicture.

FIG. 18(a) shows a case where the block BLa10 in the picture B11 isdecoded in the direct mode with reference to the picture P10 that isjust previous to the picture B11 (first example of direct modedecoding).

A base motion vector to be used for direct mode decoding of the blockBLa10 is a forward motion vector (base motion vector) MVh10 of a block(base block) BLg10 located in the same position as the block BLa10,which block BLg10 is included in the picture (base picture) P13 that isbackward referred to when decoding the block BLa10. The forward motionvector MVh10 indicates an area CRh10 corresponding to the base blockBLg10, in the picture P10 that is just previous to the picture B11.

In this case, as a forward motion vector MVk10 of the target block BLa10to be decoded, a motion vector which is parallel to the base motionvector MVh10 and indicates an area CRk10 included in the picture P10 andcorresponding to the target block BLa10, is employed. Further, as abackward motion vector MVi10 of the target block BLa10 to be decoded, amotion vector which is parallel to the base motion vector MVh10 andindicates an area CRi10 included in the picture P13 and corresponding tothe target block BLa10, is employed.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRk10 of the forward reference picture P10 as a forwardreference image, and an image in the area CRi10 of the backwardreference picture P13 as a backward reference image, from the referencepicture memory 207, and performs addition and averaging of image data ofthe both images to obtain a motion compensation image (prediction image)corresponding to the target block BLa10.

In this case, the magnitude (MVF) of the forward motion vector MVk10 andthe magnitude (MVB) of the backward motion vector MVi10 are obtained bythe above-described formulae (1) and (2), using the magnitude (MVR) ofthe base motion vector MVh10.

The magnitudes MVF and MVB of the respective motion vectors show thehorizontal component and vertical component of the motion vector,respectively.

Further, TRD indicates a time-basis distance between the backwardreference picture P13 for the target block BLa10 in the picture B11, andthe picture P10 which is forward referred to when decoding the block(base block) BLg10 in the backward reference picture (base picture) P13.Further, TRF is the time-basis distance between the target picture B11and the just-previous reference picture P10, and TRB is the time-basisdistance between the target picture B11 and the picture P10 which isreferred to when decoding the block BLg10 in the backward referencepicture P13.

FIG. 18(b) shows a case where a block BLa20 in the picture B11 isdecoded in the direct mode with reference to the picture P10 that isjust previous to the picture B11 (second example of direct modedecoding).

In this second example of direct mode decoding, in contrast with thefirst example of direct mode decoding shown in FIG. 18(a), a picturewhich is forward referred to in decoding the base block (i.e., a blockplaced in the same position as the target block, in the backwardreference picture for the target block) is the picture P7.

That is, a base motion vector to be used for direct mode decoding of theblock BLa20 is a forward motion vector MVh20 of a block BLg20 located inthe same position as the block BLa20, which block BLg20 is included inthe picture P13 that is backward referred to when decoding the blockBLa20. The forward motion vector MVh20 indicates an area CRh20corresponding to the base block BLg20, in the picture P7 that ispositioned forward the target picture B11.

In this case, as a forward motion vector MVk20 of the target block BLa20to be decoded, a motion vector, which is parallel to the base motionvector MVh20 and indicates an area CRk20 included in the picture P10 andcorresponding to the target block BLa20, is employed. Further, as abackward motion vector MVi20 of the target block BLa20 to be decoded, amotion vector, which is parallel to the base motion vector MVh20 andindicates an area CRi20 included in the picture P13 and corresponding tothe target block BLa20, is employed.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRk20 of the forward reference picture P10 as a forwardreference image, and an image in the area CRi20 of the backwardreference picture P13 as a backward reference image, from the referencepicture memory 207, and performs addition and averaging of image data ofthe both images to obtain a motion compensation image (prediction image)corresponding to the target block BLa20.

In this case, the magnitude (MVF) of the forward motion vector MVk20 andthe magnitude (MVB) of the backward motion vector MVi20 are obtained bythe above-described formulae (1) and (2), using the magnitude (MVR) ofthe base motion vector MVh20, as described for the first example ofdirect mode decoding.

FIG. 19(a) shows a case where a block BLa30 in the picture B11 isdecoded in the direct mode with reference to the picture P7 which ispositioned forward the picture P10 that is positioned just previous tothe picture B11 (third example of direct mode decoding).

In this third example of direct mode decoding, in contrast with thefirst and second examples of direct mode coding shown in FIGS. 18(a) and18(b), a picture to be forward referred to in decoding the target blockis not a picture just previous to the target picture, but a picture thatis forward referred to in decoding the base block (a block in the sameposition as the target block) in the base picture. The base picture is apicture that is backward referred to in decoding the target block.

That is, a base motion vector to be used in direct mode decoding of theblock BLa30 is a forward motion vector MVh30 of a block BLg30 located inthe same position as the block BLa30, which block BLg30 is included inthe picture P13 that is backward referred to in decoding the blockBLa30. The forward motion vector MVh30 indicates an area CRh30corresponding to the base block BLg30, in the picture P7 that ispositioned forward the target picture B11.

In this case, as a forward motion vector MVk30 of the target block BLa30to be decoded, a motion vector, which is parallel to the base motionvector MVh30 and indicates an area CRk30 included in the picture P7 andcorresponding to the target block BLa30, is employed. Further, as abackward motion vector MVi30 of the target block BLa30 to be decoded, amotion vector, which is parallel to the base motion vector MVh30 andindicates an area CRi30 included in the picture P13 and corresponding tothe target block BLa30, is employed.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRk30 of the forward reference picture P7 as a forwardreference image, and an image in the area CRi30 of the backwardreference picture P13 as a backward reference image, from the referencepicture memory 207, and performs addition and averaging of image data ofthe both images to obtain a motion compensation image (prediction image)corresponding to the target block BLa30.

In this case, the magnitude (MVF) of the forward motion vector MVk30 andthe magnitude (MVB) of the backward motion vector MVi30 are obtained bythe above-described formulae (2) and (3), using the magnitude (MVR) ofthe base motion vector MVh30.

When the picture to be referred to in decoding the block BLg30 hasalready been deleted from the reference picture memory 207, the forwardreference picture P10 that is timewise closest to the target picture isused as a forward reference picture in the third example of direct modedecoding. In this case, the third example of direct mode decoding isidentical to the first example of direct mode decoding.

FIG. 19(b) shows a case where a block BLa40 in the picture B11 isdecoded in the direct mode by using a motion vector whose magnitude iszero (fourth example of direct mode decoding).

In this fourth example of direct mode decoding, the magnitude of thereference motion vector employed in the first and second examples shownin FIGS. 18(a) and 18(b) is zero.

In this case, as a forward motion vector MVk40 and a backward motionvector MVi40 of the block BLa40 to be decoded, a motion vector whosemagnitude is zero is employed.

That is, the forward motion vector MVk40 indicates an area (block) CRk40of the same size as the target block, which area is included in thepicture P10 and placed at the same position as the target block BLa40.Further, the backward motion vector MVi40 indicates an area (block)CRi40 of the same size as the target block, which area is included inthe picture P13 and placed at the same position as the target blockBLa40.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area (block) CRk40 of the forward reference picture P10 as aforward reference image, and an image in the area (block) CRi40 of thebackward reference picture P13 as a backward reference image, from thereference picture memory 207, and performs addition and averaging ofimage data of the both images to obtain a motion compensation image(prediction image) corresponding to the target block BLa40. This methodis applicable to, for example, a case where a block which is included inthe picture P13 as a backward reference picture of the picture B11 andis located in the same position as the block BLa40 is a block having nomotion vector like an intra-frame-coded block.

The data of the motion compensation image thus generated is output tothe addition unit 208. The addition unit 208 adds the inputtedprediction error data and the motion compensation image data to generatedecoded image data. The decoded image data so generated is outputthrough the switch 210 to the reference picture memory 207, and thedecoded image is stored in the reference picture memory 207.

The memory control unit 204 controls the reference picture memory 207 onthe basis of the header information Ih indicating what kind of referencehas been carried out in coding the P pictures and B pictures extractedfrom the header information of the bit stream.

As described above, the blocks in the picture B11 are successivelydecoded. When all of the blocks in the picture B11 have been decoded,decoding of the picture B12 takes place.

In the B picture decoding described above, a specific block is sometimestreated as a skip block. Hereinafter, decoding of a skip block will bebriefly described.

When it is found that a specific block is treated as a skip block duringdecoding of an inputted bit stream, from a skip identifier or a blocknumber information that is described in the bit stream, motioncompensation, i.e., acquisition of a prediction image corresponding to atarget block, is carried out in the direct mode.

For example, as shown in FIG. 6(b), when the blocks MB(r+1) and MB(r+2)between the block MB(r) and the block MB(r+3) in the picture B11 aretreated as skip blocks, the bit stream analysis unit 201 detects theskip identifier Sf from the bit stream Bs. When the skip identifier Sfis input to the mode decoding unit 223, the mode decoding unit 223instructs the motion compensation decoding unit 205 to perform motioncompensation in the direct mode.

Then, the motion compensation decoding unit 205 obtains the predictionimages of the blocks MB(r+1) and MB(r+2), on the basis of an image(forward reference image) of a block which is included in the forwardreference picture P10 and placed in the same position as the blocktreated as a skip block, and an image (backward reference image) of ablock in the same position as the block treated as a skip block, andthen outputs the data of the prediction images to the addition unit 208.The prediction error decoding unit 202 outputs data whose value is zero,as difference data of the blocks treated as skip blocks. In the additionunit 208, since the difference data of the blocks treated as skip blocksis zero, the data of the prediction images of the blocks MB(r+1) andMB(r+2) are output to the reference picture memory 207 as decoded imagesof the blocks MB(r+1) and MB(r+2).

Furthermore, in the direct mode processing shown in FIG. 18(a) (firstexample), the direct mode processing shown in FIG. 18(b) (secondexample), and the direction mode processing shown in FIG. 19(a) (thirdexample), all of blocks whose difference data become zero are notnecessarily treated as skip blocks. That is, a target block is subjectedto bidirectional prediction using a picture that is positioned justprevious to the target picture as a forward reference picture, and amotion vector whose magnitude is zero, and only when the difference dataof the target block becomes zero, this target block may be treated as askip block.

In this case, when it is found, from the skip identifier or the like inthe bit stream Bs, that a specific block is treated as a skip block,motion compensation should be carried out by bidirectional predictionwhose motion is zero, using a just-previous reference picture as aforward reference picture.

Decoding Process for Picture B12

Since the bit stream analysis unit 201, the mode decoding unit 223, andthe prediction error decoding unit 202 operate in the same way asdescribed for decoding of the picture P10, repeated description is notnecessary.

The motion compensation decoding unit 205 generates motion compensationimage data from the inputted information such as the motion vector. Themotion vector MV and the reference picture index Rp are input to themotion compensation decoding unit 205. The picture P12 has been codedusing the pictures P7, P10 and B11 as candidate pictures for forwardreference, and the picture P13 as a candidate picture for backwardreference. At decoding the target picture, these candidate pictures havealready been decoded, and are stored in the reference picture memory207.

The timewise change of the pictures stored in the reference picturememory 207, and the method for determining a reference picture areidentical to those in the case of decoding the picture B11 describedwith respect to FIG. 3.

When the coding mode is bidirectional predictive coding, the motioncompensation decoding unit 205 determines which one of the pictures P7,P10 and B11 has been used for forward reference, from the referencepicture index. Then, the motion compensation decoding unit 205 obtains aforward reference image from the reference picture memory 207 on thebasis of the forward motion vector, and further, it obtains a backwardreference image from the reference picture memory 207 on the basis ofthe backward motion vector. Then, the motion compensation decoding unit205 performs addition and averaging of image data of the forwardreference image and the backward reference image to generated a motioncompensation image corresponding to the target block.

(Bidirectional Prediction Mode)

FIG. 20 illustrates a case where the target picture to be decoded is thepicture B12, and bidirectional predictive decoding is performed for ablock (target block) BLa02 to be decoded, in the picture B12.

Initially, a description will be given of a case where the forwardreference picture is the picture B11, and the backward reference pictureis the picture P13.

In this case, the forward motion vector is a motion vector MVe02indicating an area CRe02 in the picture B11 corresponding to the blockBLa02. The backward motion vector is a motion vector MVg02 indicating anarea CRg02 in the picture P13 corresponding to the block BLa02.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRe02 in the picture B11 as a forward reference image, andan image in the area CRg02 in the picture P13 as a backward referenceimage, from the reference picture memory 207, and performs addition andaveraging of image data of the images in the both areas CRe02 and CRg02to obtain a motion compensation image corresponding to the target blockBLa02.

Next, a description will be given of a case where the forward referencepicture is the picture P10, and the backward reference picture is thepicture P13.

In this case, the forward motion vector is a motion vector MVf02indicating an area CRf02 in the picture P10, corresponding to the blockBLa02. The backward motion vector is a motion vector MVg02 indicating anarea CRg02 in the picture P13, corresponding to the block BLa02.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRf02 in the picture P10 as a forward reference image and animage in the area CRg02 in the picture P13 as a backward reference imagefrom the reference picture memory 207, and performs addition andaveraging of image data of the images in the both areas CRf02 and CRg02to obtain a motion compensation image corresponding to the target blockBLa02.

(Direct Mode)

Further, when the coding mode is the direct mode, the motioncompensation decoding unit 205 obtains a motion vector (base motionvector) of a reference block (a block whose relative position is thesame as that of the target position) in the backward reference pictureP13 for the target picture B12, which motion vector is stored in themotion vector storage unit 226. The motion compensation decoding unit205 obtains a forward reference image and a backward reference imagefrom the reference picture memory 207 by using the base motion vector.Then, the motion compensation decoding unit 205 performs addition andaveraging of image data of the forward reference image and the backwardreference image, thereby generating a motion compensation imagecorresponding to the target block.

FIG. 21(a) shows a case where the block BLa50 in the picture B12 isdecoded in the direct mode with reference to the picture B11 that isjust previous to the picture B12 (first example of direct modedecoding).

A base motion vector to be used for direct mode decoding of the blockBLa50 is a forward motion vector MVj50 of the base block (i.e., theblock BLg50 placed in the same position as the block BLa50) in thepicture P13 that is backward referred to when decoding the block BLa50.The forward motion vector MVj50 indicates an area CRj50 corresponding tothe base block BLg50 in the picture P10 that is positioned forward andclose to the picture B11.

In this case, as a forward motion vector MVk50 of the target block BLa50to be decoded, a motion vector which is parallel to the base motionvector MVj50 and indicates an area CRk50 included in the picture B11 andcorresponding to the target block BLa50, is employed. Further, as abackward motion vector MVi50 of the target block BLa50 to be decoded, amotion vector which is parallel to the base motion vector MVj50 andindicates an area CRi50 included in the picture P13 and corresponding tothe target block BLa50, is employed.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRk50 of the forward reference picture B11 as a forwardreference image and an image in the area CRi50 of the backward referencepicture P13 as a backward reference image from the reference picturememory 207, and performs addition and averaging of image data of theboth images to obtain a motion compensation image (prediction image)corresponding to the target block BLa50.

In this case, the magnitude (MVF) of the forward motion vector MVk50 andthe magnitude (MVB) of the backward motion vector MVi50 are obtained bythe above-described formulae (1) and (2) using the magnitude (MVR) ofthe base motion vector MVh10.

The magnitudes MVF and MVB of the respective motion vectors show thehorizontal component and vertical component of the motion vector,respectively.

FIG. 21(b) shows a case where a block BLa60 in the picture B12 isdecoded in the direct mode with reference to the picture B11 that ispositioned forward the picture B12 (second example of direct modedecoding).

In this second example of direct mode decoding, in contrast with thefirst example of direct mode decoding shown in FIG. 21(a), a picturewhich is forward referred to in decoding the base block (i.e., a blockplaced in the same position as the target block, in the backwardreference picture for the target block) is the picture P7.

That is, a base motion vector to be used for direct mode decoding of theblock BLa60 is a forward motion vector MVj60 of the reference block (theblock BLg60 in the same position as the block BLa60) in the picture P13that is backward referred to when decoding the block BLa60. The forwardmotion vector MVj60 indicates an area CRj60 corresponding to the baseblock BLg60, in the picture P7 that is positioned forward the targetpicture B12.

In this case, as a forward motion vector MVk60 of the target block BLa60to be decoded, a motion vector, which is parallel to the base motionvector MVj60 and indicates an area CRk60 included in the picture B11 andcorresponding to the target block BLa60, is employed. Further, as abackward motion vector MVi60 of the target block BLa60 to be decoded, amotion vector, which is parallel to the base motion vector MVj60 andindicates an area CRi60 included in the picture P13 and corresponding tothe target block BLa60, is employed.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRk60 of the forward reference picture B11 as a forwardreference image and an image in the area CRi60 of the backward referencepicture P13 as a backward reference image from the reference picturememory 207, and performs addition and averaging of image data of theboth images to obtain a motion compensation image (prediction image)corresponding to the target block BLa60.

In this case, the magnitude (MVF) of the forward motion vector MVk60 andthe magnitude (MVB) of the backward motion vector MVi60 are obtained bythe above-described formulae (1) and (2), using the magnitude (MVR) ofthe base motion vector MVj60, as described for the first example ofdirect mode decoding.

FIG. 22(a) shows a case where a block BLa70 in the picture B12 isdecoded in the direct mode with reference to the picture P7 which ispositioned forward the forward picture P10 that is closest to thepicture B12 (third example of direct mode decoding).

In this third example of direct mode decoding, in contrast with thefirst and second examples of direct mode coding shown in FIGS. 21(a) and21(b), a picture to be forward referred to in decoding the target blockis not a picture just previous to the target picture, but a picture thatis forward referred to in decoding the base block in the base picture.The base picture is a picture that is backward referred to in decodingthe target block.

That is, a base motion vector to be used in direct mode decoding of theblock BLa70 is a forward motion vector MVj70 of a base block BLg70 (ablock in the same position as the block BLa70) in the picture P13 thatis backward referred to in decoding the block BLa70. The forward motionvector MVj70 indicates an area CRj70 corresponding to the base blockBLg70 in the picture P7 that is positioned forward the target pictureB12.

In this case, as a forward motion vector MVk70 of the target block BLa70to be decoded, a motion vector which is parallel to the base motionvector MVj70 and indicates an area CRk70 included in the picture P7 andcorresponding to the target block BLa70, is employed. Further, as abackward motion vector MVi70 of the target block BLa70, a motion vectorwhich is parallel to the base motion vector MVj70 and indicates an areaCRi70 included in the picture P13 and corresponding to the target blockBLa70, is employed.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area CRk70 of the forward reference picture P7 as a forwardreference image and an image in the area CRi70 of the backward referencepicture P13 as a backward reference image from the reference picturememory 207, and performs addition and averaging of image data of theboth images to obtain a motion compensation image (prediction image)corresponding to the target block BLa70.

In this case, the magnitude (MVF) of the forward motion vector MVk70 andthe magnitude (MVB) of the backward motion vector MVi70 are obtained bythe above-described formulae (2) and (3), using the magnitude (MVR) ofthe base motion vector MVj70.

When the picture to be referred to in decoding the block BLg70 hasalready been deleted from the reference picture memory 207, the forwardreference picture P10 that is timewise closest to the target picture isused as a forward reference picture in the third example of direct modedecoding. In this case, the third example of direct mode decoding isidentical to the first example of direct mode decoding.

FIG. 22(b) shows a case where a block BLa80 in the picture B12 isdecoded in the direct mode by using a motion vector whose magnitude iszero (fourth example of direct mode decoding).

In this fourth example of direct mode decoding, the magnitude of thereference motion vector employed in the first and second examples shownin FIGS. 21(a) and 21(b) is zero.

In this case, as a forward motion vector MVk80 and a backward motionvector MVi80 of the block BLa80 to be decoded, a motion vector whosemagnitude is zero is employed.

That is, the forward motion vector MVk80 indicates an area (block) CRk80of the same size as the target block, which area is included in thepicture B11 and placed at the same position as the target block BLa80.Further, the backward motion vector MVi80 indicates an area (block)CRi80 of the same size as the target block, which area is included inthe picture P13 and placed at the same position as the target blockBLa80.

Accordingly, the motion compensation decoding unit 205 obtains an imagein the area (block) CRk80 of the forward reference picture B11 as aforward reference image and an image in the area (block) CRi80 of thebackward reference picture P13 as a backward reference image from thereference picture memory 207, and performs addition and averaging ofimage data of the both images to obtain a motion compensation image(prediction image) corresponding to the target block BLa80. This methodis applicable to, for example, a case where a block which is included inthe picture P13 as a backward reference picture of the picture B11 andis located in the same position as the block BLa80 is a block having nomotion vector like an intra-frame-coded block.

The data of the motion compensation image thus generated is output tothe addition unit 208. The addition unit 208 adds the inputtedprediction error data and the motion compensation image data to generatedecoded image data. The decoded image data so generated is outputthrough the switch 210 to the reference picture memory 207.

As described above, the blocks in the picture B12 are successivelydecoded. The image data of the respective pictures stored in thereference picture memory 207 are rearranged in order of time to beoutput as output image data Od.

Thereafter, the pictures following the picture B12, which are arrangedin decoding order times as shown in FIG. 16(a), are successively decodedaccording to the picture type, in like manner as described for thepictures P13, B11, and B12. FIG. 16(b) shows the pictures rearranged inorder of display times.

During decoding of the inputted bit stream, if it is found that aspecific block is treated as a skip block, from a skip identifier or ablock number information that is described in the bit stream, motioncompensation, i.e., acquisition of a prediction image corresponding to atarget block, is carried out in the direct mode as in the case ofdecoding the picture B11.

As described above, in the moving picture decoding apparatus 20according to the second embodiment, when decoding a block in a Bpicture, a prediction image corresponding to the target block isgenerated using an already-decoded P picture and an already-decoded Bpicture as candidate pictures for forward reference, on the basis ofinformation (reference picture index) indicating candidate pictureswhich are forward referred to in coding the target block, whichinformation is included in the bit stream corresponding to the targetblock to be decoded. Therefore, it is possible to correctly decode ablock in a target B picture which has been coded using a B picture as acandidate picture for forward reference.

Further, in the moving picture decoding apparatus 20, when a targetblock to be decoded which is included in a B picture has been coded inthe direct mode, a motion vector of the target block is calculated onthe basis of a motion vector of a block that is placed in the sameposition as the target block. Therefore, it is not necessary for thedecoding end to obtain the information indicating the motion vector ofthe block coded in the direct mode, from the coding end.

Furthermore, in the moving picture decoding apparatus 20, the data ofthe already-decoded pictures which are stored in the reference picturememory are managed on the basis of the information indicating thecandidate pictures which are used in coding p pictures and B pictures,which information is included as header information in the bit stream.For example, at the completion of decoding one picture, data of pictureswhich are not to be used as reference pictures in decoding the followingpictures are successively deleted, whereby the picture memory can beused with efficiency.

Further, when decoding a target block in a P picture, it is possible todetermine which one of plural candidate pictures is used as a referencepicture (i.e., which one of the candidate pictures is referred to indetecting the motion vector of the target block to be decoded) from thereference picture information added to the motion vector information.

Likewise, when decoding a target block in a B picture, it is possible todetermine which one of plural candidate pictures for forward referenceis used as a reference picture (i.e., which one of the candidatepictures is referred to in detecting the forward motion vector of thetarget block to be decoded) from the reference picture information addedto the motion vector information.

While in this second embodiment the direct mode is used as one of theplural coding modes for B pictures, the direct mode is not necessarilyused as the coding mode for B pictures. In this case, the motion vectorstorage unit 226 in the moving picture decoding apparatus 20 isdispensed with.

Further, while in this second embodiment four specific methods aredescribed as examples of direct mode (i.e., the first example shown inFIG. 18(a) or 21(a), the second example shown in FIG. 18(b) or 21(b),the third example shown in FIG. 19(a) or 22(a), and the fourth exampleshown in FIG. 19(b) or 22(b)), the decoding apparatus performs decodingusing a method suited to a coding method which is used as direct mode bythe coding apparatus. More specifically, when plural methods areemployed as direct mode, the decoding apparatus performs decoding, usinginformation indicating which one of the plural methods is used asspecific direct mode, that is described in the bit stream.

In this case, the operation of the motion compensation decoding unit 205varies according to the information. For example, when this informationis added in block units for motion compensation, the mode decoding unit223 determines which one of the four methods mentioned above is used asdirect mode in coding, and notifies the motion compensation decodingunit 205 of the determined method. The motion compensation decoding unit205 performs appropriate motion compensation predictive decodingaccording to the determined method of direct mode.

Further, when the information (DM mode information) indicating which oneof the plural methods is used as direct mode is described in the headerof the entire sequence, the GOP header, the picture header, or the sliceheader, the DM mode information is transferred for every sequence, GOP,picture, or slice, from the bit stream analysis unit 201 to the motioncompensation decoding unit 205, and the motion compensation decodingunit 205 changes the operation.

While in this second embodiment two B pictures are placed between an Ipicture and a P picture or between adjacent P pictures, the number ofcontinuous B pictures may be three or four.

Further, while in this second embodiment three pictures are used ascandidate pictures for a forward reference picture for a P picture, thenumber of reference candidate pictures for a P picture may be other thanthree.

Furthermore, while in this second embodiment two I or P pictures and oneB picture are used as candidate pictures for a forward reference picturein decoding a B picture, forward reference candidate pictures indecoding a B picture are not restricted thereto.

Moreover, in this second embodiment, as a method for managing thereference picture memory in decoding the picture P13, picture B11, andpicture B12, a method of collectively managing the P pictures and Bpictures to be used as candidates of a reference picture, as shown inFIG. 3, is described. However, the reference picture memory managingmethod may be any of the four methods which are described for the firstembodiment with reference to FIGS. 11 to 14, wherein all of the picturesto be used as candidates for a reference picture are separated into Ppictures and B pictures to be managed.

In this case, the reference picture memory 207 has memory areas for sixpictures, i.e., P picture memory areas (#1)˜(#4), and B picture memoryareas (#1) and (#2). Further, these six memory areas are not necessarilyformed in one reference picture memory, but each of the six memory areasmay be constituted by one independent reference picture memory.

Further, when the coding end employs a reference picture index assigningmethod wherein it is determined, for each picture to be coded, which ofthe P picture memory area and the B picture memory area is givenpriority in assigning reference picture indices as shown in FIG. 14, themoving picture decoding apparatus can easily identify a picture which isused as a reference picture among plural candidate pictures, on thebasis of the reference picture indices, by using information describedin the bit stream, which indicates the memory area taking priority.

For example, when the target picture to be decoded is the picture B11,since the forward reference picture that is timewise closest to thetarget picture is the picture P10, reference picture indices areassigned to the pictures stored in the P picture memory with priority.Accordingly, a reference picture index [0] is added as headerinformation to the bit stream of the target block when the picture P10is used as a reference picture in coding the target block of the pictureB11. Likewise, a reference picture index [1] is added as headerinformation when the picture P7 is used as a reference picture, and areference picture index [2] is added as header information when thepicture B9 is used as a reference picture. Accordingly, the movingpicture decoding apparatus can know which candidate picture is used as areference picture in coding the target block, according to the referencepicture index.

In this case, since information indicating that reference pictureindices are assigned to the candidate pictures in the P picture memorywith priority is included as header information in the bit stream,identification of the reference picture is further facilitated by usingthis information.

Further, when the target picture to be decoded is the picture B12, sincethe forward reference picture that is timewise closest to the targetpicture is the picture B11, reference picture indices are assigned tothe pictures stored in the B picture memory with priority. Accordingly,a reference picture index [0] is added as header information to the bitstream of the target block when the picture B11 is used as a referencepicture in coding the target block of the picture B12. Likewise, areference picture index [1] is added as header information when thepicture P10 is used as a reference picture, and a reference pictureindex [2] is added as header information when the picture P7 is used asa reference picture. Accordingly, the moving picture decoding apparatuscan know which candidate picture is used as a reference picture incoding the target block, according to the reference picture index.

In this case, since information indicating that reference pictureindices are assigned to the candidate pictures in the B picture memorywith priority is included as header information in the bit stream,identification of the reference picture is further facilitated by usingthis information.

Furthermore, there are cases where, at the coding end, one of theabove-mentioned five methods for managing the reference picture memory(refer to FIGS. 3, 11 to 14) is previously selected, or some of thesefive methods are used by switching them. For example, when the codingend employs some of the plural methods by switching them, the movingpicture decoding apparatus can determine the reference picture index,according to information indicating which method is used for eachpicture, that is described in the bit stream.

Furthermore, in this second embodiment, the five methods for managingthe reference picture memory (refer to FIGS. 3, 11 to 14) are describedfor the case where there are three reference candidate pictures for a Ppicture, and there are two P pictures and one B picture as forwardreference candidate pictures for a B picture. However, the five methodsfor managing the reference picture memory are also applicable to caseswhere the number of reference candidate pictures is different to thosementioned above. When the number of reference candidate pictures isdifferent from those mentioned for the second embodiment, the capacityof the reference picture memory is also different from that describedfor the second embodiment.

Moreover, in this second embodiment, in the method of managing thereference picture memory wherein the stored reference candidates areseparated into P pictures and B pictures (four examples shown in FIGS.11 to 14), the P pictures are stored in the P picture memory area whilethe B pictures are stored in the B picture memory area. However, ashort-term picture memory and a long-term picture memory which aredefined in H.263++ may be used as memory areas where pictures arestored. For example, the short-term picture memory and the long-termpicture memory may be used as a P picture memory area and a B picturememory area, respectively.

Embodiment 3

FIG. 23 is a block diagram illustrating a moving picture codingapparatus 30 according to a third embodiment of the present invention.

The moving picture coding apparatus 30 can switch, according to acontrol signal supplied from the outside, a method for assigningreference picture indices to candidate pictures, between a method ofassigning reference picture indices to candidate pictures according toan initialized rule (default assignment method), and an adaptiveassignment method of assigning reference picture indices to candidatepictures by the default assignment method and, further, adaptivelychanging the assigned reference picture indices according to the codingstatus.

To be specific, one operation mode of the moving picture codingapparatus 30 according to the third embodiment is the operation of themoving picture coding apparatus 10 according to the first embodiment. Inother words, when the default assignment method is selected as areference picture index assignment method of the moving picture codingapparatus 30, the moving picture coding apparatus 30 performs the sameprocessing as that of the moving picture coding apparatus 10.

Hereinafter, the moving picture coding apparatus 30 will be described indetail.

The moving picture coding apparatus 30 is provided with a coding controlunit 130, instead of the coding control unit 110 of the moving picturecoding apparatus 10 according to the first embodiment. The codingcontrol unit 130 switches, according to an external control signal Cont,a method for assigning reference picture indices to candidate pictures,between a method of assigning reference picture indices according to aninitialized rule (default assignment method), and a method including afirst step of assigning reference picture indices to candidate picturesby the default assignment method, and a second step of adaptivelychanging the reference picture indices which are assigned to thecandidate pictures by the default assignment method (adaptive assignmentmethod).

Further, the coding control unit 130 includes a detection unit (notshown) which detects, for every target picture to be coded, codingefficiency in a case where each of plural reference candidate picturesis used as a reference picture. The coding control unit 130 changes thereference picture index which is assigned to each candidate picture bythe default assignment method, according to the coding efficiencydetected by the detection unit.

More specifically, the coding control unit 130 changes the referencepicture index which is assigned to each candidate picture by the defaultassignment method, such that, among plural candidate pictures for atarget picture, a candidate picture which provides a higher codingefficiency of the target picture when it is used as a reference pictureis given a smaller reference picture index.

Then, the mode selection unit 139 selects, in the direct mode, a picturethat is assigned a reference picture index [0], as a forward referencepicture for a target block. In a predictive coding mode other than thedirect mode such as the bidirectional predictive coding mode, the modeselection unit 139 selects a reference picture from among pluralcandidate pictures according to the coding efficiency.

Other components of the moving picture coding apparatus 30 according tothe third embodiment are identical to those of the moving picture codingapparatus 10 according to the first embodiment.

Hereinafter, the operation will be described.

In the moving picture coding apparatus 30, when the default assignmentmethod is selected as a method for assigning reference picture indicesto candidate pictures according to the external control signal Cont, theoperation of the moving picture coding apparatus 30 is identical to theoperation of the moving picture coding apparatus 10 according to thefirst embodiment.

On the other hand, when the adaptive assignment method is selected as amethod for assigning reference picture indices to candidate picturesaccording to the external control signal Cont, the moving picture codingapparatus 30 performs, in the first step, assignment of referencepicture indices in like manner as described for the moving picturecoding apparatus 10.

When the adaptive assignment method is selected, the moving picturecoding apparatus 30 performs, in the second step, adaptive change of thereference picture indices that are assigned by the default assignmentmethod.

Hereinafter, a description will be given of specific methods ofassigning reference picture indices in the case where the adaptiveassignment method is selected. In the following description, it isassumed that a target picture is the picture B12.

Initially, in the first step, as shown in FIG. 3, reference pictureindices are assigned to candidate pictures for forward reference suchthat a smaller reference picture index is assigned to a candidatepicture that is closer to the target picture. That is, a referencepicture index [1] is assigned to the reference picture P10, a referencepicture index [0] is assigned to the reference picture B11, and areference picture index [2] is assigned to the reference picture P7.

Next, in the second step, as shown in FIG. 24, the reference pictureindex [1] of the reference picture P10 is changed to [0], and thereference picture index [0] of the reference picture B11 is changed to[1].

Such rewriting of reference picture indices is carried out for everytarget picture, according to the coding efficiency. Further, the movingpicture coding apparatus 30 inserts information indicating which of thedefault assignment method and the adaptive assignment method is set asan assignment method, as header information, in the bit stream. Further,when the adaptive assignment method is set, information indicating howthe assignment of reference picture indices is carried out is alsoinserted as header information in the bit stream.

As described above, in this third embodiment, the reference pictureindex of the candidate picture which is to be used as a forwardreference picture in the direction mode, can be changed to [0].

That is, since, in the first embodiment, a smaller reference pictureindex is given to a reference candidate picture that is timewise closerto the target picture, only the picture B11 that is timewise closest tothe target picture B12 can be referred to in the direct mode. In thisthird embodiment, however, any picture other than the picture B11closest to the target picture B12 can be used as a forward referencepicture, if the coding efficiency is improved.

Further, in this case, since the picture to be referred to in coding thepicture B12 in the direct mode is not the picture B11 but the pictureB10, decoding of the picture B11 becomes unnecessary. Accordingly, asshown in FIG. 25(a), a B picture immediately after a P picture can beprocessed without decoding it, whereby speedup of decoding is achievedwhen the picture B11 is not necessary. Further, since decoding can becarried out even when the data of the picture B11 is lost due totransmission error or the like, reliability of decoding is improved.

As described above, when a reference picture index can be arbitrarilyassigned to a candidate picture to intentionally determine a picture tobe referred to in the direct mode, a predetermined picture can beprocessed without decoding it, as shown in FIG. 25(a).

Furthermore, even when three B pictures are placed between P pictures asshown in FIG. 25(b), a predetermined picture can be processed withoutdecoding it. Therefore, when a picture that is not needed by the user ispreviously known at the coding end, such picture can be omitted toreduce the processing time in decoding.

In FIG. 25(b), even when the picture B3 is not decoded, other picturescan be decoded.

That is, in the assignment method of the first embodiment, since thepicture B4 refers to the picture B3 in the direct mode, the picture B3must be decoded to decode the picture B4. In this third embodiment,however, since a picture to be referred to in the direct mode can bearbitrarily set, decoding of the picture B3 can be dispensed with.

Furthermore, in this third embodiment, assignment of reference pictureindices is carried out such that a smaller reference picture index isassigned to a candidate picture that is timewise closer to the targetpicture, and a reference picture to be used in the direct mode isdetermined according to the reference picture indices. Therefore, thecoding efficiency can be improved by a reduction in the motion vector,and further, the processing time can be reduced.

Furthermore, when the target block is processed in the direct mode atthe decoding end, since the forward reference candidate picture to whichthe reference picture index [0] is assigned is immediately used as areference picture, decoding time can be reduced.

Furthermore, while in this third embodiment a candidate picture whosereference picture index should be changed to [0] is determined accordingto the coding efficiency, a reference picture index of a picture whichis most likely to be referred to, e.g., a P picture that is timewiseclosest to the target picture, may be changed to [0].

Moreover, while in this third embodiment a picture to be referred to inthe direct mode is a picture whose reference picture index is [0], thepresent invention is not restricted thereto. For example, informationindicating that a picture is to be referred to in the direct mode iscoded, and decoding may be carried out in the direct mode on the basisof this information.

Embodiment 4

FIG. 26 is a block diagram for explaining a moving picture decodingapparatus 40 according to a fourth embodiment of the present invention.

The moving picture decoding apparatus 40 receives the bit streamoutputted from the moving picture coding apparatus 30 of the thirdembodiment, and performs decoding of each picture, on the basis ofinformation indicating which of the default assignment method and theadaptive assignment method should be used when assigning referencepicture indices (assignment method instruction information), whichinformation is included in the bit stream.

That is, one operation mode of the moving picture decoding apparatus 40according to the fourth embodiment is the operation of the movingpicture coding apparatus 20 according to the second embodiment. In otherwords, when the default assignment method is used as a reference pictureindex assignment method in the moving picture decoding apparatus 40, theoperation of the moving picture decoding apparatus 40 is identical tothat of the moving picture decoding apparatus 20.

Hereinafter, the moving picture decoding apparatus 40 will be describedin detail.

The moving picture decoding apparatus 40 is provided with a memorycontrol unit 244, instead of the memory control unit 204 of the movingpicture decoding apparatus 20 according to the second embodiment. Thememory control unit 244 performs memory management according to eitherthe default assignment method or the adaptive assignment method, on thebasis of the assignment method instruction information included in thebit stream as header information.

Other components of the moving picture decoding apparatus 40 accordingto the fourth embodiment are identical to those of the moving picturedecoding apparatus 20 according to the second embodiment.

Hereinafter, the operation will be described.

The moving picture decoding apparatus 40 operates in accordance with theassignment method instruction information that is included as headerinformation in the bit stream supplied from the moving picture codingapparatus 30.

That is, when the default assignment method is selected as a referencepicture index assignment method at the coding end, i.e., wheninformation indicating that the default assignment method is selected isincluded in the bit stream, the moving picture decoding apparatus 40operates in the same manner as the moving picture decoding apparatus 20of the second embodiment.

On the other hand, when the adaptive assignment method is selected as areference picture index assignment method at the coding end, i.e., wheninformation indicating that the adaptive assignment method is selectedis included in the bit stream, the moving picture decoding apparatus 40operates in accordance with the adaptive assignment method. In thiscase, since information indicating how the assignment of referencepicture indices is carried out is also included as header information inthe bit stream, assignment of reference picture indices is carried outaccording to this information.

Hereinafter, a description will be given of the operation of the movingpicture decoding apparatus 40 in the case where the adaptive assignmentmethod is selected.

In the reference picture memory 207, as shown in FIG. 24, referencecandidate pictures stored in the respective memory area are rewrittenevery time a target picture is processed.

To be specific, when the target picture to be decoded is the pictureB12, decoding of a target block in the picture B12 is carried out withreference to a reference picture that is selected from candidatepictures according to the header information of the target block.

For example, when the coding mode for the target block is thebidirectional predictive mode, a candidate picture which is given thesame reference picture index as the reference picture index that isincluded in the header information of the target block, is selected as aforward reference picture from among the candidate pictures P10, B11,and P7. When the reference picture index included in the headerinformation of the target block is [1], the candidate picture B11 isselected as a forward reference picture. Then, the target block issubjected to bidirectional predictive decoding with reference to thecandidate picture B11 as a forward reference picture, and the pictureP13 as a backward reference picture.

Further, when the decoding mode of the target block is the direct mode,a candidate picture (picture P10) which is given the reference pictureindex [0] is selected as a forward reference picture from among thecandidate pictures P7, P10, and B9. Then, the target block is decodedwith reference to the candidate picture P10 as a forward referencepicture, and the picture P13 as a backward reference picture.

As described above, according to the fourth embodiment, the referencepicture memory 207 is managed as shown in FIG. 24, that is, memorymanagement is carried out using, as the reference picture indices of therespective candidate pictures, those obtained by changing the referencepicture indices assigned by the default assignment method, according tothe coding status. Therefore, it is possible to realize a decodingmethod adaptive to a coding method in which the reference pictureindices of the candidate pictures are rewritten according to the codingefficiency.

That is, since, in the second embodiment, a smaller reference pictureindex is given to a reference candidate picture that is timewise closerto the target picture, only the picture B11 that is timewise closest tothe target picture B12 can be used as a reference picture in the directmode. In this fourth embodiment, however, a picture other than thepicture B11 closest to the target picture B12 can be used as a forwardreference picture.

Further, in this case, since the picture to be referred to in decoding ablock in the picture B12 in the direct mode is not the picture B11 butthe picture B10, decoding of the picture B11 becomes unnecessary.Accordingly, as shown in FIG. 25(a), a B picture immediately after a Ppicture can be processed without decoding it, whereby speedup ofdecoding is achieved when the picture B11 is not necessary. Further,since decoding can be carried out even when the data of the picture B11is lost due to transmission error or the like, reliability of decodingis improved.

As described above, when a reference picture index to be assigned toeach reference candidate picture is arbitrarily selected according tothe coding status to intentionally determine a picture to be referred toin the direct mode, a predetermined picture can be processed withoutdecoding it as shown in FIG. 25(a).

Furthermore, even when three B pictures are placed between P pictures asshown in FIG. 25(b), a predetermined picture can be processed withoutdecoding it. Therefore, if a picture that is not needed by the user ispreviously known at the coding end, such picture can be omitted toreduce the processing time for decoding.

In FIG. 25(b), even when the picture B3 is not decoded, other picturescan be decoded.

That is, since, in the second embodiment, the picture B4 is decoded withreference to the picture B3 in the direct mode, the picture B3 must bedecoded. In this fourth embodiment, however, since a picture to bereferred to in the direct mode is arbitrarily set at the coding end,decoding of the picture B3 can be dispensed with.

Furthermore, when the target block is processed in the direct mode atthe decoding end, since the forward reference candidate picture to whichthe reference picture index [0] is assigned is immediately used as areference picture, decoding time can be reduced.

While in the first to fourth embodiments a B picture is not referred towhen coding or decoding a P picture, a B picture may be referred to whencoding or decoding a P picture.

Further, while in the first to fourth embodiments a time-basis distancebetween pictures is calculated according to the display times of therespective pictures, it may be calculated according to information otherthan time information such as the display times of pictures.

For example, a counter value that is incremented every time a picture isprocessed is set, and a time-basis distance between pictures may becalculated according to this count value.

To be specific, when time information is included in both of a videostream and an audio stream corresponding to a single contents, it is noteasy to manage video data and audio data on the basis of the timeinformation so as to maintain synchronization between these data,because a unit of time information is small. However, managementconsidering synchronization between video data and audio data isfacilitated by managing arrangement of the respective pictures with thecounter value.

Furthermore, in the first to fourth embodiments, a header section and adata section in a data processing unit, such as a GOP or a picture, arenot separated from each other, and they are included in a bit streamcorresponding to each data processing unit to be transferred. However,the header section and the data section may be separated from each otherto be transferred in different streams.

For example, when a stream is transferred in units of data transfer suchas packets into which the stream is divided, a header section and a datasection corresponding to a picture may be transferred separately fromeach other. In this case, the header section and the data section arenot always included in the same stream. However, in data transfer usingpackets, even when the header section and the data section are notcontinuously transferred, the corresponding header section and datasection are merely transferred in different packets, and therelationship between the corresponding header section and data sectionis stored in header information of each packet, and therefore, it issubstantially identical to that the header section and the data sectionare included in the same bit stream.

Furthermore, while in the first to fourth embodiments the referencepicture indices are used as information for identifying which one ofplural reference candidate pictures is referred to in coding a targetblock, the reference picture indices may be used as informationindicating the positions of plural forward reference candidate picturesfor a target picture to be coded or decoded. To be specific, in thereference picture index assignment methods according to the first andsecond embodiments or the default assignment methods according to thethird and fourth embodiments, reference picture indices are assigned tothe plural forward reference candidate pictures such that a smallerreference picture index is assigned to a candidate picture closer to thetarget picture, and therefore, the position of each forward referencecandidate picture (i.e., the ordinal rank of each forward referencecandidate picture in nearness to the target picture, among all forwardreference candidate pictures) can be detected according to the referencepicture index assigned to the forward reference candidate picture.

Furthermore, position identification information indicating thepositions of the respective pictures constituting a moving picture onthe display time axis may be included in the bit stream corresponding tothe moving picture, separately from the reference picture indicesindicating the relative positions of the forward reference candidatepictures. The position identification information is different from thetime information indicating the display times of pictures, and it isinformation specifying the relative positions of the respectivepictures.

Moreover, in the first to fourth embodiments, a picture that is to bebackward referred to when coding a block in a target picture to be codedor decoded (backward reference picture for a target picture) is used asa base picture in the direct mode. However, a base picture to be used inthe direct mode may be an already-processed picture other than thebackward reference picture for the target picture, e.g., a picture to beforward referred to when coding the block in the target picture.

Embodiment 5

FIG. 27 is a block diagram for explaining a moving picture codingapparatus 50 according to a fifth embodiment of the present invention.

The moving picture coding apparatus 50 according to the fifth embodimentis different from the moving picture coding apparatus 10 according tothe first embodiment in candidate pictures for forward referencepictures to be referred to when coding a P picture and a B picture, andcoding modes for a B picture.

That is, the moving picture coding apparatus 50 is provided with,instead of the control unit 110 and the mode selection unit 109according to the first embodiment, a control unit 150 and a modeselection unit 159 which operate in different manners from thosedescribed for the first embodiment.

To be specific, the control unit 150 according to the fifth embodimentcontrols a reference picture memory 117 in such a manner that, whencoding a P picture, four pictures (I or P pictures) which are positionedforward the P picture are used as candidate pictures for forwardreference, and when coding a B picture, four pictures (I or P pictures)which are positioned forward the B picture, a forward B picture that isclosest to the B picture, and a backward I or P picture are used ascandidate pictures.

Further, when coding a block (target block) in a P picture, the modeselection unit 159 according to the fifth embodiment selects, as acoding mode for the target block, one from among the intra-picturecoding, the inter-picture predictive coding using a motion vector, andthe inter-picture predictive coding using no motion vector (a motion istreated as zero). When coding a block (target block) in a B picture, themode selection unit 159 selects, as a coding mode for the target block,one from among the intra-picture coding, the inter-picture predictivecoding using a forward motion vector, the inter-picture predictivecoding using backward motion vector, and the inter-picture predictivecoding using a forward motion vector and a backward motion vector. Thatis, the mode selection unit 159 of the moving picture coding apparatus50 according to this fifth embodiment is different from the modeselection unit 109 of the moving picture coding apparatus 10 accordingto the first embodiment only in that it does not use the direct mode,and therefore, the moving picture coding apparatus 50 does not have themotion vector storage unit 116 of the moving picture coding apparatus10.

Further, the moving picture coding apparatus 50 according to the fifthembodiment is identical to the moving picture coding apparatus 10according to the first embodiment except the coding control unit 150 andthe mode selection unit 159.

Next, the operation will be described.

Input pictures are stored in the input picture memory 101, in units ofpictures in order of display times. As shown in FIG. 29(a), inputpictures P0, B1, B2, P3, B4, B5, P6, B7, B8, P9, B10, B11, P12, B13,B14, P15, B16, B17, and P18 are stored in the input picture memory 101in order of display times.

The respective pictures stored in the input picture memory 101 arerearranged in coding order as shown in FIG. 29(b). This rearrangement iscarried out according to the relationships between target pictures andreference pictures during inter-picture predictive coding. That is,rearrangement of the input pictures is carried out such that a secondpicture to be used as a candidate for a reference picture when coding afirst picture should be coded prior to the first picture.

In this fifth embodiment, when coding a P picture (target picture), fourpictures (I or P pictures) which are positioned timewise forward andclose to the target picture are used as candidates for a referencepicture. Further, when coding a B picture, four pictures (I or Ppictures) which are positioned timewise forward and close to the targetpicture, a B picture which is positioned timewise forward and closest tothe target picture, and an I or P picture which is positioned timewisebackward and closest to the target picture, are used as candidates for areference picture.

The respective pictures rearranged in the input picture memory 101 areread out for each unit of motion compensation. In this fifth embodiment,the unit of motion compensation is a rectangle area (macroblock) inwhich pixels are arranged in matrix, having a size of 16 pixels in thehorizontal direction×16 pixels in the vertical direction. In thefollowing description, a macroblock is simply referred to as a block.

Hereinafter, coding processes for the pictures P15, B13, and B14 will bedescribed in this order.

Coding Process for Picture P15

Since the picture P15 is a P picture, this picture is subjected tointer-picture predictive coding using forward reference. Further, incoding a P picture, no B picture is used as a reference picture.

FIG. 28 shows the manner of picture management in the reference picturememory 117.

For example, at start of coding the picture P15, in the referencepicture memory 117, the pictures P12, B11, P9, P6, and P3 are stored inmemory areas to which logical memory numbers are assigned, in ascendingorder of the logical memory numbers. These pictures have already beencoded, and the image data stored in the reference picture memory 117 areimage data which have been decoded in the moving picture codingapparatus 50. Hereinafter, for simplification, a picture whose imagedata is stored in the memory is referred to as a picture stored in thememory.

The reference candidate pictures stored in the reference picture memory117 are assigned reference picture indices under control of the codingcontrol unit 150. The assignment of reference picture indices is carriedout not in order of picture coding but in order of display times. To bespecific, a smaller reference picture index is assigned to a newerreference candidate picture, i.e., a reference candidate picture whichis later in display order. However, in coding a P picture, no referencepicture indices are assigned to B pictures. Further, in coding a Bpicture, a newest reference candidate picture is assigned a code [b]indicating that this picture should be treated as a backward referencepicture.

According to the above-mentioned reference picture index determiningmethod, as shown in FIG. 28, reference picture indices [0], [1], [2],and [3] are assigned to the pictures P12, P9, P6, and P3, respectively,and no reference picture index is assigned to the picture B11.

By the way, in coding a P picture, the coding control unit 150 controlsthe respective switches so that the switches 113, 114, and 115 areturned ON. A block in the picture P15 that is read from the inputpicture memory 101 is input to the motion vector detection unit 108, themode selection unit 109, and the difference calculation unit 102.

The motion vector detection unit 108 detects a motion vector of theblock in the picture P15, using the pictures P12, P9, P6, and P3 towhich the reference picture indices are assigned, among the picturesstored in the input picture memory 117. In this case, an optimumreference candidate picture is selected from among the pictures P12, P9,P6, and P3, and detection of the motion vector is carried out withreference to the selected reference picture. Then, the detected motionvector is output to the mode selection unit 159 and the bit streamgeneration unit 104. Further, information Rp indicating which one of thepictures P12, P9, P6, and P3 is referred to in detecting the motionvector, i.e., the reference picture index, is also output to the modeselection unit 159.

The mode selection unit 159 determines a coding mode for the block inthe picture P15, using the motion vector detected by the motion vectordetection unit 108. The coding mode indicates a method for coding theblock. For example, for a block in a P picture, a coding mode isselected from among the intra-picture coding, the inter-picturepredictive coding using a motion vector, and the inter-picturepredictive coding using no motion vector (i.e., motion is regarded as0). Generally, selection of a coding mode is carried out so that codingerror at a predetermined amount of bits is minimized.

The coding mode Ms determined by the mode selection unit 159 is outputto the bit stream generation unit 104. Further, when the determinedcoding mode is the coding mode which performs forward reference, thereference picture index is also output to the bit stream generation unit104.

Further, a prediction image Pd which is obtained on the basis of thecoding mode determined by the mode selection unit 152 is output to thedifference calculation unit 102 and the addition unit 106. However, whenthe intra-picture coding is selected, no prediction image Pd isoutputted. Further, when the intra-picture coding is selected, theswitch 111 is controlled so that the input terminal Ta is connected tothe output terminal Tb2, and the switch 112 is controlled so that theoutput terminal Td is connected to the input terminal Tc2.

Hereinafter, a description will be given of a case where theinter-picture predictive coding is selected in the mode selection unit109. Since the operations of the difference calculation unit 102,prediction error coding unit 103, bit stream generation unit 104, andprediction error decoding unit 105 are identical to those mentioned forthe first embodiment, repeated description is not necessary.

When coding of all blocks in the picture P15 is completed, the codingcontrol unit 150 updates the logical memory numbers and the referencepicture indices corresponding to the pictures stored in the referencepicture memory 117.

That is, since the coded picture P15 is later in the order of displaytimes than any pictures stored in the reference picture memory 117, thepicture P15 is stored in the memory area in which the logical memorynumber (0) is set. Then, the logical memory numbers of the memory areaswhere other reference pictures have already been stored are incrementedby 1. Further, since the next target picture to be coded is the pictureB13 that is a B picture, a reference picture index is also assigned tothe picture B11. Thereby, the pictures P15, P12, B11, P9, P6, and P3 arestored in the memory areas in which the logical memory numbers (0)˜(5)are set, respectively, and the reference picture indices [0], [1], [2],[3], and [4] are assigned to the pictures P12, B11, P9, P6, and P3,respectively. Since the next target picture is a B picture, the pictureP15 stored in the memory area of the logical memory number 0 is assigneda code [b] indicating that this picture is treated as a backwardreference picture, instead of the reference picture index.

Coding Process for Picture B13

Since the picture B13 is a B picture, this picture is subjected tointer-picture predictive coding using bidirectional reference. In thiscase, four I or P pictures which are timewise close to the targetpicture and a B picture which is timewise closest to the target pictureare used as candidate pictures for forward reference, and an I or Ppicture which is timewise closest to the target picture is used as acandidate picture for backward reference. Accordingly, the candidatepictures for forward reference for the picture B13 are the pictures P12,B11, P9, P6, and P3, and the candidate picture for backward referencefor the picture B13 is the picture P15. These reference candidatepictures are stored in the reference picture memory 117. These referencecandidate pictures are assigned logical memory numbers and referencepicture indices as shown in FIG. 28.

In coding a B picture, the coding control unit 150 controls therespective switches so that the switches 113, 114, and 115 are turnedON. Accordingly, a block in the picture B11 that is read from the inputpicture memory 101 is input to the motion vector detection unit 108, themode selection unit 109, and the difference calculation unit 102.

The motion vector detection unit 108 detects a forward motion vector anda backward motion vector of the block in the picture B13, using thepictures P12, B11, P9, P6, and P3 stored in the reference picture memory117, as candidate pictures for forward reference, and the picture P15 asa candidate picture for backward reference. In this case, an optimumpicture is selected from among the pictures P12, B11, P9, P6, and P3,and detection of the forward motion vector is carried out with referenceto the selected picture. Then, the detected motion vector is output tothe mode selection unit 159 and the bit stream generation unit 104.Further, information Rp indicating which one of the pictures P12, B11,P9, P6, and P3 is referred to in detecting the forward motion vector,i.e., the reference picture index, is also output to the mode selectionunit 159.

The operations of the mode selection unit 150, difference calculationunit 102, bit stream generation unit 104, and prediction error decodingunit 105 are identical to those for coding the picture P15.

When coding of all blocks in the picture B13 is completed, the codingcontrol unit 150 updates the logical memory numbers and the referencepicture indices corresponding to the pictures stored in the referencepicture memory 117.

That is, since the picture B13 is positioned, in order of display times,before the picture P15 stored in the reference picture memory 117 andafter the picture P12 stored in the reference picture memory 17, thepicture B13 is stored in the memory area in which the logical memorynumber (1) is set. Further, since the picture B11 is not used as areference picture in coding the subsequent pictures, the picture B11 isdeleted. At this time, information indicating that the picture B11 isdeleted from the reference picture memory is output to the bit streamgeneration unit 104 as a control signal Cs1. The bit stream generationunit 104 describes this information as header information in the bitstream. Further, the logical memory number of the memory areacorresponding to the picture P12 is incremented by 1.

The next target picture to be coded is the picture B14 as a B picture.Accordingly, the picture stored in the memory area with the logicalmemory number (0) is used as a backward reference picture, and referencepicture indices are assigned to the other pictures. Thereby, thepictures P15, B13, P12, P9, P6, and P3 are stored in the memory areascorresponding to the logical memory numbers (0)˜(5), respectively, andthe reference picture indices [0], [1], [2], [3], and [4] are assignedto the pictures B13, P12, P9, P6, and P3, respectively.

Coding Process for Picture B14

Since the picture B14 is a B picture, this picture is subjected tointer-picture predictive coding using bidirectional reference. Asreference pictures for the picture B14, the pictures B13, P12, P9, P6,and P3 are used as forward reference pictures while the picture P15 isused as a backward reference picture. In processing a B picture, thecoding control unit 150 controls the respective switches so that theswitches 113, 114, and 115 are turned ON. Accordingly, a block in thepicture B14 that is read from the input picture memory 101 is input tothe motion vector detection unit 108, the mode selection unit 109, andthe difference calculation unit 102.

The motion vector detection unit 108 detects a forward motion vector anda backward motion vector of the block in the picture B14, using thepictures B13, P12, P9, P6, and P3 stored in the reference picture memory117 as candidate pictures for forward reference as well as the pictureP15 as a candidate picture for backward reference. In this case, anoptimum picture is selected from among the pictures B13, P12, P9, P6,and P3, and detection of the forward motion vector is carried out withreference to the selected picture. Then, the detected motion vector isoutput to the mode selection unit 159 and the bit stream generation unit104. Further, information Rp indicating which one of the pictures B13,P12, P9, P6, and P3 is referred to in detecting the forward motionvector, i.e., the reference picture index, is also output to the modeselection unit 159.

The operations of the mode selection unit 150, difference calculationunit 102, bit stream generation unit 104, prediction error decoding unit105, and addition unit 106 are similar to those for coding the pictureP15.

When coding of all blocks in the picture B14 is completed, the codingcontrol unit 150 updates the logical memory numbers and the referencepicture indices corresponding to the pictures stored in the referencepicture memory 117.

That is, since the picture B14 is positioned, in order of display times,before the picture P15 stored in the reference picture memory 117, andlater than the picture B13 stored in the reference picture memory 117,the picture B14 is stored in the memory area in which the logical memorynumber (1) is set. Further, since the picture B13 is not used as areference picture in coding the subsequent pictures, the picture B13 isdeleted. At this time, information indicating that the picture B13 isdeleted from the reference picture memory is output to the bit streamgeneration unit 104 as a control signal Cd1. The bit stream generationunit 104 describes this information as header information in the bitstream.

The next target picture to be coded is the picture P18 that is a Ppicture. Accordingly, reference picture indices are assigned to thepictures other than B pictures. Thereby, the pictures P15, B14, P12, P9,and P6 are stored in the memory areas corresponding to the logicalmemory numbers (0)˜(5), respectively, and the reference picture indices[0], [1], [2], and [3] are assigned to the pictures P15, P12, P9, andP6, respectively.

As described above, according to the fifth embodiment, plural candidatepictures for forward reference for a target picture to be coded areassigned reference picture indices such that a smaller index is assignedto a candidate picture whose display time is later (i.e., informationfor identifying which one of the candidate pictures is used in detectingthe forward motion vector of the target block). Therefore, a candidatepicture which is most likely to be selected as a reference picture amongthe plural candidate pictures is assigned a smaller reference pictureindex. Accordingly, the amount of codes of the reference picture indicescan be minimized, resulting in an increase in coding efficiency.

Hereinafter, the effects of this fifth embodiment will be describedtaking a case where coding of a B picture is carried out using another Bpictures as a reference candidate picture, together with the problems ofthe prior art.

For example, it is assumed that pictures of a moving picture arearranged in display order as shown in FIG. 29(a), and four P picturesand one B picture are used as candidate pictures for forward referencein coding a target picture.

FIG. 30 shows an example of management of pictures stored in thereference picture memory. The candidate pictures are stored in codingorder, in the memory.

When coding the picture P15, in the reference picture memory, thepictures B11, P12, P9, P6, and P3 are stored in the memory areas, inascending order of the logical memory numbers. Further these candidatepictures are assigned the reference picture indices [0], [1], [2], [3],and [4], respectively. Therefore, a reference picture index is assignedto a B picture (picture B11 in this case) which is not used as areference picture in coding a P picture, and the reference picture indexnot to be used causes degradation in coding efficiency.

Further, when coding the picture B13, in the reference picture memory,the pictures P15, B11, P12, P9, P6, and P3 are stored in the memoryareas, in ascending order of the logical memory numbers. The picture P15is assigned a code [b] indicating that this picture is used as abackward reference picture, and the remaining pictures are assigned thereference picture indices [0], [1], [2], [3], and [4], respectively.Therefore, the reference picture index assigned to the picture B11 thatis timewise far from the picture B13 (target picture) is smaller thanthe reference picture index assigned to the picture P12 that is timewiseclose to the target picture B13. In performing motion detection,generally, a candidate picture that is timewise closer to a targetpicture is more likely to be used as a reference picture. Accordingly,when the reference picture index of the picture B11 that is far from thetarget picture is smaller than the reference picture index of thepicture P12 that is close to the target picture, coding efficiency isdegraded.

Furthermore, when coding the picture B14, in the reference picturememory, the pictures B13, P15, B11, P12, P9, and P6 are stored in thememory areas, in ascending order of the logical memory numbers. Thepicture B13 is assigned a code [b] indicating that this picture is usedas a backward reference picture, and the remaining pictures are assignedthe reference picture indices [0], [1], [2], [3], and [4], respectively.Therefore, the picture P15 which should actually be used as a candidatepicture for backward reference for the picture B14, is used as acandidate picture for forward reference. Moreover, the picture B13 whichshould actually be used as a candidate picture for forward reference forthe picture B14, is used as a candidate picture for backward reference.As a result, it becomes difficult to perform correct coding. Further, incoding the picture B14, the picture B11 which is not used as a referencepicture exists in the reference picture memory.

On the other hand, according to the fifth embodiment of the invention,as shown in FIG. 28, the reference candidate pictures for the targetpicture are stored in display order in the reference picture memory, andthe candidate pictures for forward reference are assigned the referencepicture indices such that a candidate picture whose display time islater is assigned a smaller reference picture index, and therefore, acandidate picture which is more likely to be selected as a referencepicture from among the candidate pictures is assigned a smallerreference picture index. Thereby, the amount of codes of the referencepicture indices can be minimized, resulting in an increase in codingefficiency.

Further, since, in coding a P picture, no reference picture indices areassigned to B pictures, occurrence of reference picture indices thatwill never be used is avoided, resulting in a further increase in codingefficiency.

Moreover, when coding a B picture, no reference picture index isassigned to the picture that is stored in the memory area correspondingto the smallest logical memory number, and this picture is used as abackward reference picture. Therefore, in predictive coding of a Bpicture, a P picture to be used as a backward reference picture isprevented from being used as a forward reference picture.

Further, when a picture that is not used as a reference picture isdeleted from the reference picture memory, information indicating thisdeletion is described in the bit stream. Therefore, the decoding end candetect that the picture which is not to be used as a reference picturein decoding a target picture and the following pictures, is deleted fromthe reference picture memory.

In this fifth embodiment, motion compensation is performed in units ofimage spaces (macroblocks) each comprising 16 pixels in the horizontaldirection×16 pixels in the vertical direction, and coding of aprediction error image is performed in units of image spaces (subblocks)each comprising 8 pixels in the horizontal direction×8 pixels in thevertical direction. However, the number of pixels in each macroblock(subblock) in motion compensation (coding of a prediction error image)may be different from that described for the fifth embodiment.

Further, while in this fifth embodiment the number of continuous Bpictures is two, the number of continuous B pictures may be three ormore.

Further, while in this fifth embodiment four pictures are used ascandidate pictures for a forward reference picture in coding a Ppicture, the number of forward reference candidate pictures for a Ppicture may be other than four.

Furthermore, while in this fifth embodiment four P pictures and one Bpicture are used as candidate pictures for a forward reference picturein coding a B picture, forward reference candidate pictures for a Bpicture are not restricted thereto.

Furthermore, in this fifth embodiment, each of plural picturesconstituting a moving picture, which is a target picture to be coded, isused as a reference picture when coding another picture that follows thetarget picture. However, the plural pictures constituting a movingpicture may include pictures not to be used as reference pictures. Inthis case, the pictures not to be used as reference pictures are notstored in the reference picture memory, whereby the same effects asdescribed for the fifth embodiment can be achieved.

Furthermore, while in this fifth embodiment coding of a B picture iscarried out using another B picture as a reference candidate picture,coding of a B picture may be carried out without referring to another Bpicture. In this case, no B pictures are stored in the reference picturememory. Also in this case, the same effects as described for the fifthembodiment can be achieved by assigning reference picture indicesaccording to the order of picture display times.

Furthermore, while in this fifth embodiment a single system of referencepicture indices are assigned, different systems of reference pictureindices may be assigned in the forward direction and the backwarddirection, respectively.

Moreover, while in this fifth embodiment a smaller reference pictureindex is assigned to a candidate picture for forward reference whosedisplay time is later, the reference picture index assignment method isnot restricted thereto so long as a smaller reference picture index isassigned to a candidate picture that is more likely to be selected as areference picture.

FIG. 31 is a conceptual diagram illustrating the structure of a bitstream (format of a coded image signal) corresponding to pictures towhich reference picture indices are assigned.

A coded signal Pt equivalent to one picture includes header informationHp placed at the beginning of the picture, and a data section Dp thatfollows the header information Hp. The header information Hp includes acontrol signal (RPSL). The data section Dp includes coded data (bitstream) corresponding to each block.

For example, a bit stream BLx is a bit stream of a block that is codedin intra-picture coding mode, and a bit stream BLy is a bit stream of ablock that is coded in inter-picture predictive coding mode other thanintra-picture coding mode.

The block bit stream BLx includes header information Hbx, informationPrx relating to a coding mode, and coded image information Dbx. Theblock bit stream BLy includes header information Hby, information Pryrelating to a coding mode, first reference picture index R1 d 1, asecond reference picture index R1 d 2, a first motion vector MV1, asecond motion vector MV2, and coded image information Dby. Which of thefirst and second reference picture indices R1 d 1 and R1 d 2 should beused is determined from the information Pry relating to the coding mode.

A reference picture index R1 d 1 is assigned to a forward referencecandidate picture with priority over a backward reference candidatepicture. A reference picture index R1 d 2 is assigned to a backwardreference candidate picture with priority over a forward referencecandidate picture.

Embodiment 6

FIG. 32 is a block diagram for explaining a moving picture decodingapparatus 60 according to a sixth embodiment of the present invention.

The moving picture decoding apparatus 60 according to the sixthembodiment decodes the bit stream Bs outputted from the moving picturecoding apparatus 50 according to the fifth embodiment.

The moving picture decoding apparatus 60 is different from the movingpicture decoding apparatus 20 according to the second embodiment incandidate pictures for forward reference pictures to be referred to whencoding a P picture and a B picture, and coding modes for a B picture.

That is, the moving picture decoding apparatus 60 is provided with,instead of the memory control unit 204 and the mode decoding unit 223according to the second embodiment, a memory control unit 264 and a modedecoding unit 263 which operate in different manners from thosedescribed for the second embodiment.

To be specific, the memory control unit 264 according to the sixthembodiment controls a reference picture memory 207 such that, whendecoding a P picture, four pictures (I or P pictures) which arepositioned forward the P picture are used as candidate pictures forforward reference, and when decoding a B picture, four pictures (I or Ppictures) which are positioned forward the B picture, a forward Bpicture that is closest to the B picture, and a backward I or P pictureare used as candidate pictures.

Further, when decoding a block (target block) in a P picture, the modedecoding unit 263 according to the sixth embodiment selects, as a codingmode for the target block, one from among plural modes as follows:intra-picture decoding, inter-picture predictive decoding using a motionvector, and inter-picture predictive decoding using no motion vector (amotion is treated as zero). When decoding a block (target block) in a Bpicture, the mode decoding unit 263 selects, as a decoding mode for thetarget block, one from among plural modes as follows: intra-picturedecoding, inter-picture predictive decoding using a forward motionvector, inter-picture predictive decoding using backward motion vector,and inter-picture predictive decoding using a forward motion vector anda backward motion vector.

That is, the mode decoding unit 263 of the moving picture decodingapparatus 60 according to this sixth embodiment is different from themode decoding unit 223 of the moving picture decoding apparatus 20according to the second embodiment only in that it does not use adecoding process corresponding to the direct mode, and therefore, themoving picture decoding apparatus 60 does not have the motion vectorstorage unit 226 of the moving picture decoding apparatus 20.

Further, the moving picture decoding apparatus 60 according to the sixthembodiment is identical to the moving picture decoding apparatus 20according to the second embodiment except the memory control unit 264and the mode decoding unit 263.

Next, the operation of the moving picture decoding apparatus 60 will bedescribed.

The bit stream Bs outputted from the moving picture coding apparatus 50according to the fifth embodiment is input to the moving picturedecoding apparatus 60 shown in FIG. 32. In the bit stream Bs, each Ppicture has been subjected to inter-picture predictive coding, usingfour I or P pictures which are positioned timewise forward and close tothe P picture, as reference candidate pictures. Further, each B picturehas been coded using four P pictures which are positioned timewiseforward and closest to the B picture, a B picture which is positionedtimewise forward the B picture, and an I or P picture which ispositioned timewise backward and closest to the B picture.

In this case, the order of the pictures in the bit stream is as shown inFIG. 29(b).

Hereinafter, decoding processes for the pictures P15, B13, and B14 willbe described in this order.

Decoding Process for Picture P15

The bit stream of the picture P15 is input to the bit stream analysisunit 201. The bit stream analysis unit 201 extracts various kinds ofdata from the inputted bit stream Bs. The various kinds of data areinformation such as a coding mode, a motion vector, and the like. Theextracted information for mode selection (coding mode) Ms is output tothe mode decoding unit 263. Further, the extracted motion vector MV isoutput to the motion compensation decoding unit 205. Furthermore, theprediction error coded data Ed is output to the prediction errordecoding unit 202.

The mode decoding unit 263 controls the switches 209 and 210 withreference to the coding mode Ms extracted from the bit stream. When thecoding mode is inter-picture coding, the switch 209 is controlled suchthat the input terminal Te is connected to the output terminal Tf1, andthe switch 210 is controlled such that the output terminal Th isconnected to the input terminal Tg1. When the coding mode isinter-picture predictive coding, the switch 209 is controlled such thatthe input terminal Te is connected to the output terminal Tf1, and theswitch 210 is controlled such that the output terminal Th is connectedto the input terminal Tg2.

Further, the mode decoding unit 263 outputs the coding mode Ms also tothe motion compensation decoding unit 205.

Hereinafter, a description will be given of the case where the codingmode is inter-picture predictive coding.

The prediction error decoding unit 202 decodes the inputted coded dataEd to generate prediction error data PDd. The generated prediction errordata PDd is output to the switch 209. Since the input terminal Te of theswitch 209 is connected to the output terminal Tf1, the prediction errordata PDd is output to the addition unit 208.

The motion compensation decoding unit 205 generates a motioncompensation image from the inputted information such as the motionvector. The information inputted to the motion compensation decodingunit 205 is the motion vector MV and the reference picture index Rp. Themotion compensation decoding unit 205 obtains a motion compensationimage (prediction image) from the reference picture memory 207, on thebasis of the inputted information. The picture P15 has been coded usingthe pictures P12, P9, P6, and P3 as candidates for a reference picture,and these candidate pictures have already been decoded and are stored inthe reference picture memory 207.

FIG. 28 shows the pictures stored in the reference picture memory 207.As shown in FIG. 28, when decoding the picture P15, the pictures P12,B11, P9, P6, and P3 are stored in the reference picture memory 207.

The memory control unit 264 assigns reference picture indices to thereference candidate pictures stored in the reference picture memory 117.This assignment of reference picture indices is carried according to theorder of picture display times such that a smaller reference pictureindex is assigned to a newer reference candidate picture. In decoding aP picture, no reference picture indices are assigned to B pictures.Accordingly, reference picture indices [0], [1], [2], and [3] areassigned to the pictures P12, P9, P6, and P3, respectively, and noreference picture index is assigned to the picture B11.

The motion compensation decoding unit 205 determines which one of thepictures P12, P9, P6, and P3 is used as a reference picture when codingthe target block, from the reference picture indices. Then, the motioncompensation decoding unit 205 obtains a prediction image (predictiondata Pd) from the reference picture memory 207 on the basis of thedetermined reference picture and the motion vector to generate a motioncompensation image. The motion compensation image so generated is inputto the addition unit 208.

The addition unit 208 adds the prediction error data PDd and the motioncompensation image to generate a decoded image (data Ad). The decodedimage so generated is output through the switch 210 to the referencepicture memory 207.

When all of the macroblocks in the picture P15 have been decoded, thememory control unit 264 updates the logical memory numbers and thereference picture indices corresponding to the pictures stored in thereference picture memory 207.

At this time, since, in order of time, the picture P15 is later than anypictures stored in the reference picture memory 117, the picture P15 isstored in the memory area in which the logical memory number (0) is set.Then, the logical memory numbers of the memory areas where otherreference pictures have already been stored are incremented by 1.

Further, since the next target picture to be decoded is the picture B13,a reference picture index is assigned to the picture B11. Thereby, thepictures P15, P12, B11, P9, P6, and P3 are stored in the memory areas inwhich the logical memory numbers (0)˜(5) are set, respectively, and thereference picture indices [0], [1], [2], [3], and [4] are assigned tothe pictures P12, B11, P9, P6, and P3, respectively.

Decoding Process for Picture B13

Since the operations of the bit stream analysis unit 201, the modedecoding unit 203, and the prediction error decoding unit 202 areidentical to those described for decoding of the picture P15, repeateddescription is not necessary.

The motion compensation decoding unit 205 generates a motioncompensation image from the inputted information such as the motionvector. The information inputted to the motion compensation decodingunit 205 is the motion vector and the reference picture index. Thepicture B13 has been coded using the pictures P12, B11, P9, P6, and P3as candidate pictures for forward reference, and the picture P15 as acandidate picture for backward reference. At decoding of the pictureB13, these candidate pictures have already been decoded and are storedin the reference picture memory 207.

When the coding mode is forward predictive coding or bidirectionalpredictive coding, the motion compensation decoding unit 205 determineswhich one of the candidate pictures P12, B11, P9, P6, and P3 is used asa forward reference picture when coding the picture B13, on the basis ofthe reference picture indices. Then, the motion compensation decodingunit 205 obtains a forward motion compensation image from the referencepicture memory 207 on the basis of the determined reference picture andthe motion vector. When the coding mode is bidirectional predictivecoding or backward predictive coding, the motion compensation decodingunit 205 obtains a backward motion compensation image from the referencepicture memory 207 on the basis of the determined reference picture andthe backward motion vector. Then, the motion compensation decoding unit205 generates a motion compensation image (prediction picture) using theforward motion compensation image and the backward motion compensationimage.

The motion compensation image so generated is output to the additionunit 208. The addition unit 208 adds the inputted prediction error imageand motion compensation image to generate a decoded image. The decodedimage so generated is output through the switch 210 to the referencepicture memory 207.

When all of the blocks in the picture B13 have been decoded, the memorycontrol unit 264 updates the logical memory numbers and the referencepicture indices corresponding to the pictures stored in the referencepicture memory 207. Since the picture B13 is forward the picture P15stored in the reference picture memory 207 in the order of display timesand it is later than the picture P12 stored in the reference picturememory 207, the picture B13 is stored in the memory area in which thelogical memory number (1) is set.

Further, information indicating that the picture B11 is to be deletedfrom the reference picture memory is described in the bit stream, thememory control unit 264 controls the reference picture memory 207 so asto delete the picture B11 from the memory.

Further, the logical memory number of the memory area where the otherreference candidate picture P12 is stored is incremented by 1. Thereby,the pictures P15, B13, P12, P9, P6, and P3 are stored in the memoryareas in which the logical memory numbers (0)˜(5) are set, respectively,and the reference picture indices [0], [1], [2], [3], and [4] areassigned to the pictures B13, P12, P9, P6, and P3, respectively.

Decoding Process for Picture B14

Since the operations of the bit stream analysis unit 201, the modedecoding unit 203, and the prediction error decoding unit 202 areidentical to those described for decoding of the picture P15, repeateddescription is not necessary.

The motion compensation decoding unit 205 generates a motioncompensation image from the inputted information such as the motionvector. The information inputted to the motion compensation decodingunit 205 is the motion vector and the reference picture index. Thepicture B14 has been coded using the pictures B13, P12, P9, P6, and P3as candidate pictures for forward reference, and the picture P15 as acandidate picture for backward reference. At decoding of the pictureB14, these candidate pictures have already been decoded and are storedin the reference picture memory 207.

When the coding mode is forward predictive coding or bidirectionalpredictive coding, the motion compensation decoding unit 205 determineswhich one of the candidate pictures B13, P12, P9, P6, and P3 is used asa forward reference picture when coding the picture B14, on the basis ofthe reference picture indices. Then, the motion compensation decodingunit 205 obtains a forward motion compensation image from the referencepicture memory 207 on the basis of the determined reference picture andthe forward motion vector. When the coding mode is bidirectionalpredictive coding or backward predictive coding, the motion compensationdecoding unit 205 obtains a backward motion compensation image from thereference picture memory 207 on the basis of the determined referencepicture and the backward motion vector. Then, the motion compensationdecoding unit 205 generates a motion compensation image, using theforward motion compensation image and the backward motion compensationimage.

The motion compensation image so generated is output to the additionunit 208. The addition unit 208 adds the inputted prediction error imageand motion compensation image to generate a decoded image. The decodedimage so generated is output through the switch 210 to the referencepicture memory 207.

When all of the blocks in the picture B14 have been decoded, the memorycontrol unit 264 updates the logical memory numbers and the referencepicture indices corresponding to the pictures stored in the referencepicture memory 207. Since the picture B14 is forward the picture P15stored in the reference picture memory 207 in the order of display timesand it is later than the picture B13 stored in the input picture memory207, the picture B14 is stored in the memory area in which the logicalmemory number (1) is set. Further, since information indicating that thepicture B13 is to be deleted from the reference picture memory isdescribed in the bit stream, the memory control unit 264 controls thereference picture memory 207 so as to delete the picture B13 from thememory.

Since the next target picture to be decoded is the picture P18 that is aP picture, reference picture indices are assigned to pictures other thanB pictures. Thereby, the pictures P15, B14, P12, P9, and P6 are storedin the memory areas in which the logical memory numbers (0)˜(5) are set,respectively, and the reference picture indices [0], [1], [2], [3], and[4] are assigned to the pictures P15, P12, P9, and P6, respectively.

Furthermore, the decoded pictures are outputted from the referencepicture memory 207, as output images arranged in order of display times.

Thereafter, the subsequent pictures are similarly decoded according tothe picture type.

As described above, according to the sixth embodiment, reference pictureindices are assigned to plural candidate pictures for forward referencefor a target picture to be decoded such that a smaller reference pictureindex is assigned to a candidate picture whose display time is later(i.e., information for identifying which candidate picture is referredto in detecting a forward motion vector of a target block), and areference picture is determined from among the plural candidate pictureson the basis of the reference picture indices included in the bit streamof the target picture. Therefore, a smaller reference picture index isassigned to a candidate picture that is more likely to be used as areference picture. Accordingly, it is possible to correctly decode a bitstream which is obtained by a highly-efficient coding method that canminimize the amount of codes corresponding to the reference pictureindices.

Further, since, in decoding a P picture, no reference picture indicesare assigned to B pictures, it is possible to correctly decode a bitstream which is obtained by a highly-efficient coding method that canavoid occurrence of reference picture indices which will never be used.

Furthermore, since, in decoding a B picture, a picture stored in amemory area on which a smallest logic memory number is set is used as abackward reference picture and no reference picture index is assigned tothis picture, it is possible to correctly decode a bit stream which isobtained by a highly-efficient coding method that can prevent a Ppicture from being used as a forward reference picture in predictivecoding of a B picture.

Moreover, when information indicating that a picture which will never beused as a reference picture is deleted from the reference picturememory, is described in the bit stream, the reference picture is deletedfrom the reference picture memory according to the information, wherebythe reference picture memory can be effectively used.

Further, in this sixth embodiment, as an arrangement of plural picturesconstituting a moving picture, an arrangement of pictures in which two Bpictures are placed between adjacent P pictures. However, the number ofB pictures placed between adjacent P pictures may be other than two, forexample, it may be three or four.

Further, while in this sixth embodiment four pictures are used ascandidate pictures for forward reference for a P picture, the number offorward reference candidate pictures for a P picture may be other thanfour.

While in this sixth embodiment four P pictures and one B picture areused as candidate pictures for forward reference for a B picture,forward reference candidate pictures for a B picture are not restrictedthereto.

While in this sixth embodiment each of plural pictures constituting amoving picture is used as a reference picture when decoding anotherpicture that follows this picture, plural pictures constituting a movingpicture, which are to be decoded, may include pictures which will neverbe used as reference pictures. In this case, the pictures useless asreference pictures are not stored in the reference picture memory,whereby the same effects as described for the sixth embodiment can beachieved.

While in this sixth embodiment decoding of a B picture is carried outusing another B picture as a reference candidate picture, decoding of aB picture may be carried out without referring to another B picture. Inthis case, no B pictures are stored in the reference picture memory.Also in this case, the same effects as described for the sixthembodiment can be achieved by assigning reference picture indicesaccording to the order of picture display times.

While in this sixth embodiment, for simplification, a memory formanaging reference candidate pictures, and a memory for rearrangingdecoded pictures in display order to output them are not separated butdescribed as a single reference picture memory, the moving picturedecoding apparatus 60 may be provided with a management memory formanaging reference candidate pictures, and a rearrangement memory forrearranging decoded pictures in display order, respectively.

In this case, the management memory is controlled by the memorycontroller 264, and outputs reference candidate pictures to the motioncompensation decoding unit 205. Further, the rearrangement memoryrearranges the decoded pictures arranged in decoding order, in displayorder, and sequentially outputs the pictures.

Further, in this sixth embodiment, assignment of reference pictureindices to candidate pictures is carried out according to a single rule,i.e., one system of reference picture indices are used. However, twosystems of reference picture indices may be used, as described for thefifth embodiment.

Embodiment 7

FIG. 33 is a block diagram for explaining a moving picture codingapparatus 70 according to a seventh embodiment of the present invention.

This moving picture coding apparatus 70 is different from the movingpicture coding apparatus 10 according to the first embodiment incandidate pictures for forward reference pictures to be referred to whencoding a P picture and a B picture, and coding modes for a B picture.

That is, the moving picture coding apparatus 70 is provided with,instead of the control unit 110 and the mode selection unit 109according to the first embodiment, a coding control unit 170 and a modeselection unit 109 which operate in different manners from thosedescribed for the first embodiment.

To be specific, the coding control unit 170 according to the seventhembodiment controls a reference picture memory 117 such that, whencoding a P picture, three pictures (I or P pictures) which arepositioned forward the P picture are used as candidate pictures forforward reference, and when coding a B picture, two pictures (I or Ppictures) which are positioned forward the B picture, a forward Bpicture that is closest to the B picture, and a backward I or P pictureare used as candidate pictures. However, a B picture, which ispositioned forward an I or P picture that is positioned forward andclosest to the target picture, is not referred to.

The coding control unit 170 controls the bit stream generation unit 104with a control signal Cd so that a flag indicating whether or not atarget picture is to be referred to when coding subsequent pictures isinserted in the bit stream. To be specific, the code generation unit 104is controlled with the control signal Cd so that information indicatingthat data of the target picture should be stored in the referencepicture memory 117 at decoding as well as information indicating aperiod of time for the storage are added to the bit stream.

Furthermore, when coding a block (target block) in a P picture, the modeselection unit 109 according to the seventh embodiment selects, as acoding mode for the target block, one from among plural modes asfollows: intra-picture coding, inter-picture predictive coding using amotion vector, and inter-picture predictive coding using no motionvector (a motion is treated as zero). When coding a block (target block)in a B picture, the mode selection unit 179 selects, as a coding modefor the target block, one from among plural modes as follows:intra-picture coding, inter-picture predictive coding using a forwardmotion vector, inter-picture predictive coding using backward motionvector, and inter-picture predictive coding using a forward motionvector and a backward motion vector. That is, the mode selection unit179 of the moving picture coding apparatus 70 according to this seventhembodiment is different from the mode selection unit 109 of the movingpicture coding apparatus 10 according to the first embodiment only inthat it does not use the direct mode, and therefore, the moving picturecoding apparatus 70 does not have the motion vector storage unit 116 ofthe moving picture coding apparatus 10. Other constituents of the movingpicture coding apparatus 70 according to the seventh embodiment areidentical to those of the moving picture coding apparatus 10 accordingto the first embodiment.

The moving picture coding apparatus 70 according to the seventhembodiment is different from the moving picture decoding apparatus 50according to the fifth embodiment in that the coding control unit 170controls the bit stream generation unit 104 so that a flag indicatingwhether or not a target picture is to be referred to when codingsubsequent pictures is inserted in the bit stream. To be specific, thecode generation unit 104 is controlled with the control signal Cd sothat a flag indicating whether or not a target picture is to be referredto when coding subsequent pictures is inserted in the bit streamcorresponding to the target picture. Further, the moving picture codingapparatus 70 is different from the moving picture coding apparatus 50 incandidate pictures to be referred to in coding a P picture and a Bpicture. The moving picture coding apparatus 70 is identical to themoving picture coding apparatus 50 in aspects other than those mentionedabove.

Next, the operation of the moving picture coding apparatus 70 will bedescribed.

Input image data Id are stored into the input picture memory 101, inunits of pictures, in order of time.

FIG. 34(a) shows the order of pictures inputted to the input picturememory 101.

As shown in FIG. 34(a), the respective pictures are successivelyinputted to the input picture memory 101, starting from a picture P1. InFIG. 34(a), pictures P1, P4, P7, P10, P13, P16, P19, and P22 are Ppictures while pictures B2, B3, B5, B6, B8, B9, B11, B12, B14, P15, B17,P18, B20, and B21 are B pictures.

When coding a P picture, three pictures (I or P pictures) which aretimewise forward and close to the P picture are used as candidates for areference picture. Further, when coding a B picture, two pictures (I orP pictures) which are timewise forward and close to the B picture, one Bpicture that is forward and closest to the B picture, and an I or Ppicture that is forward the B picture, are used as candidates for areference picture. However, in coding a B picture, a B picture which ispositioned forward an I or P picture that is timewise forward andclosest to the B picture is not referred to. When coding an I picture,other pictures are not referred to.

The data Id of the respective pictures inputted to the input picturememory 101 are rearranged in coding order. Thereinafter the data of eachpicture is referred to simply as a picture.

That is, the process of changing the order of the pictures from inputorder to coding order is carried out on the basis of the relationshipsbetween target pictures and reference pictures in inter-picturepredictive coding. In the rearrangement, the respective pictures arerearranged so that a second picture to be used as a candidate for areference picture in coding a first picture is coded prior to the firstpicture.

To be specific, the correspondences between the pictures P10˜P13 and thereference candidate pictures are shown by arrows in FIG. 34(a). That is,when coding the P picture P10, the pictures P1, P4, and P7 are referredto, and when coding the P picture P13, the pictures P4, P7, and P10 arereferred to. Further, when coding the B picture B11, the pictures P7,P10, and P13 are referred to, and when coding the B picture B12, thepictures P7, P10, B11, and P13 are referred to.

FIG. 34(b) shows the order of the pictures after rearranging thepictures B2 to P22 shown in FIG. 34(a). After the rearrangement, therespective pictures are arranged in order of P4, B2, B3, P7, B5, B6,P10, B8, B9, P13, B11, B12, P16, B14, B15, P19, B17, B18, and p22.

The respective pictures rearranged in the reference picture memory 101are successively read out, for each predetermined data processing unit,in order of coding times. In this seventh embodiment, the dataprocessing unit is a unit of data on which motion compensation iscarried out and, more specifically, it is a rectangle image space(macroblock) in which 16 pixels are arranged in both the horizontaldirection and the vertical direction. In the following description, amacroblock is simply referred to as a block.

Hereinafter, coding processes for the pictures P13, B11, and B12 will bedescribed in this order.

Coding Process for Picture P13

Since the picture P13 is a P picture, inter-picture predictive codingusing forward reference is carried out as a coding process for thepicture P13. In this case, three I or P pictures which are positionedforward the target picture (picture P13) are used as reference candidatepictures, and specifically, the pictures P4, P7, and P10 are used. Thesereference candidate pictures have already been coded, and thecorresponding to decoded image data Dd are stored in the referencepicture memory 117.

In coding a P picture, the coding control unit 170 controls therespective switches so that the switches 113, 114, and 115 are turnedON.

Data Md corresponding to a block in the picture P13, which is read fromthe input picture memory 101, is input to the motion vector detectionunit 108, the mode selection unit 179, and the difference calculationunit 102.

The motion vector detection unit 108 detects the motion vector MV of theblock in the picture P13, using the decoded image data Rd of thepictures P4, P7, and P10 stored in the reference picture memory 117. Inthis case, an optimum picture is selected from among the pictures P4 P7,and P10, and detection of the motion vector is carried out withreference to the selected picture. Then, the detected motion vector MVis output to the mode selection unit 179 and the bit stream generationunit 104. Further, information indicating which one of the pictures P4,P7, and P10 is referred to in detecting the motion vector MV (referencepicture information) is also output to the mode selection unit 179.

The mode selection unit 179 determines a coding mode for the block inthe picture P13, using the motion vector detected by the motion vectordetection unit 108.

To be specific, in the case of coding a P picture, a coding mode isselected from among the following coding modes: intra-picture coding,inter-picture predictive coding using a motion vector, and aninter-picture predictive coding using no motion vector (i.e., motion isregarded as 0). In determining a coding mode, generally, a coding modewhich minimizes coding errors when a predetermined amount of bits isgiven to the block as an amount of codes, is selected.

The coding mode Ms determined by the mode selection unit 179 is outputto the bit stream generation unit 104. Further, when the determinedcoding mode Ms is the coding mode which performs forward reference,information indicating which one of the pictures P4, P7, and P10 isreferred to in detecting the forward motion vector (forward motionvector) is also output to the bit stream generation unit 104.

Then, prediction image data Pd, which is obtained from the referencepicture according to the coding mode Ms that is determined by the modeselection unit 179, is output to the difference calculation unit 102 andthe addition unit 106. However, when the intra-picture coding mode isselected, no prediction image data Pd is outputted. Further, when theintra-picture coding is selected, the switches 111 and 112 arecontrolled in the same manner as described for the fifth embodiment.

Hereinafter, a description will be given of a case where theinter-picture predictive coding mode is selected as the coding mode Ms.

The difference calculation unit 102, the prediction error coding unit103, the bit stream generation unit 104, the prediction error decodingunit 105, and the coding control unit 170 are identical to thosedescribed for the fifth embodiment.

However, in this seventh embodiment, information indicating that thepicture P13 is coded using forward three I or P pictures as referencecandidate pictures, is added as header information of the picture P13.Further, since the picture P13 will be referred to when coding anotherpicture, information (flag) indicating that decoded data Ddcorresponding to the picture P13 should be stored in the referencepicture memory 117 at decoding, is also added as header information ofthe picture P13. Further, information indicating that the picture P13should be stored in the reference picture memory until decoding of thepicture P22 is completed, is also added as header information of thepicture P13.

The storage period for the picture P13 may be indicated by timeinformation of the picture P22 (e.g., time-basis positional informationsuch as a picture number, decoding time information, or display timeinformation), or period information from the picture P13 to the pictureP22 (e.g., the number of pictures). The header information describedabove may be described as header information in picture units, i.e., asheader information for every target picture to be coded. Alternatively,it may be described as header information of the entire sequence, or asheader information in units of frames (e.g., in units of GOPs in MPEG).

When the coding mode for each block in the picture P13 is one performingforward reference, information indicating which one of the pictures P4,P7, and P10 is referred to in detecting the forward motion vector(reference picture information) is also added to the bit stream. Forexample, when the motion vector is obtained with reference to thepicture P10, information indicating that the P picture just previous tothe target picture is used as a reference picture (reference pictureindex) is added to the bit stream. When the motion vector is obtainedwith reference to the picture P7, information indicating that the Ppicture two-pictures previous to the target picture is used as areference picture (reference picture index) is added to the bit stream.When the motion vector is obtained with reference to the picture P4,information indicating that the P picture three-pictures previous to thetarget picture is used as a reference picture (reference picture index)is added to the bit stream. For example, a reference picture index [0]may be used to indicate that the P picture just previous to the targetpicture is used as a reference picture, a reference picture index [1]may be used to indicate that the P picture two-pictures previous to thetarget picture is used as a reference picture, and a reference pictureindex [2] may be used to indicate that the P picture three-picturesprevious to the target picture is used as a reference picture.

Further, information indicating that the P picture is subjected tointer-picture predictive coding using three reference candidate picturesis described as header information.

The remaining macroblocks in the picture P13 are coded in like manner asdescribed above. When all of the macroblocks in the picture P13 havebeen coded, coding of the picture B11 takes place.

Coding Process for Picture B11

Since the picture B11 is a B picture, inter-picture predictive codingusing bidirectional reference is carried out as a coding process for thepicture B11. In this case, two pictures (I or P pictures) which aretimewise close to the target picture (picture B11) and a B picture whichis timewise closest to the target picture are used as candidate picturesfor forward reference, and an I or P picture which is timewise closestto the target picture is used as a candidate picture for backwardreference. However, a B picture which is positioned beyond an I or Ppicture closest to the target picture is never be referred to.

Accordingly, the pictures P7 and P10 are used as forward referencepictures for the picture B11, and the picture P13 is used as a backwardreference picture for the picture B11. In processing a first B picturebetween continuous two B pictures, since this first B picture is used asa reference picture in coding the other B picture, the coding controlunit 170 controls the respective switches so that the switches 113, 114,and 115 are turned ON. Accordingly, the image data Md corresponding tothe block in the picture B11, which is read from the input picturememory 101, is input to the motion vector detection unit 108, the modeselection unit 179, and the difference calculation unit 102.

The motion vector detection unit 108 detects a forward motion vector anda backward motion vector corresponding to the target block in thepicture B11, with reference to the pictures P7 and P10 stored in thereference picture memory 117, as candidate pictures for forwardreference, and the picture P13 stored in the reference picture memory117, as a backward reference picture. In this case, either the pictureP7 or the picture P10 is selected as a most suitable reference picture,and detection of a forward motion vector is carried out according to theselected picture. The detected motion vectors are output to the modeselection unit 179 and the bit stream generation unit 104. Further,information indicating which one of the pictures P7 and P10 is referredto in detecting the forward motion vector (reference pictureinformation) is also output to the mode selection unit 179.

The mode selection unit 179 determines a coding mode for the targetblock in the picture B11, using the motion vectors detected by themotion vector detection unit 108. As a coding mode for the B picture,one of the following coding modes is selected: intra-picture codingmode, inter-picture predictive coding mode using a forward motionvector, inter-picture predictive coding mode using a backward motionpicture, and inter-picture predictive coding mode using bidirectionalmotion vectors. Also in this case, a general method (mode) whichminimizes coding errors when a predetermined amount of bits are given asthe amount of codes, should be selected.

The coding mode determined by the mode selection unit 179 is output tothe bit stream generation unit 104. Further, prediction image data Pd,which is obtained from the reference picture according to the codingmode Ms that is determined by the mode selection unit 179, is output tothe difference calculation unit 102 and the addition unit 106. However,when the intra-picture coding mode is selected by the mode selectionunit 179, no prediction image data Pd is outputted. Further, when theintra-picture coding is selected, the switches 111 and 112 arecontrolled in the same manner as described for the coding process of thepicture P13.

Hereinafter, a description will be given of a case where theinter-picture predictive coding is selected by the mode selection unit179.

In this case, the operations of the difference calculation unit 102, theprediction error coding unit 103, the bit stream generation unit 104,the prediction error decoding unit 105, and the coding control unit 170are identical to those described for the fifth embodiment.

When the coding mode is one performing forward reference, informationindicating which one of the pictures P7 and P10 is referred to indetecting the forward motion vector (reference picture information) isalso added to the bit stream. For example, when picture P10 is referredto, reference picture information indicating that a candidate picturejust previous to the target picture is used as a reference picture isadded to the bit stream. When the picture P7 is referred to, referencepicture information indicating that a candidate picture two-picturesprevious to the target picture is used as a reference picture is addedto the bit stream. For example, a reference picture index [0] may beused to indicate that a candidate picture just previous to the targetpicture is used as a reference picture, and a reference picture index[1] may be used to indicate that a candidate picture two-picturesprevious to the target picture is used as a reference picture.

Further, in this case, information indicating that the target B pictureis subjected to inter-picture predictive coding using a forward Bpicture as a reference picture is not added as header information.Furthermore, information indicating that the forward reference candidatepictures for the target B picture are two I or P pictures and one Bpicture is added as header information. Moreover, information indicatingthat a B picture, which is positioned forward an I or P picture that ispositioned forward and closest to the target B picture, is not referredto is added as header information.

Thereby, it is possible to know the capacity of a reference picturememory that is needed in decoding the bit stream Bs generated in themoving picture coding apparatus 70 according to the seventh embodiment.The header information described above may be described as headerinformation in units of pictures, i.e., as header information for everytarget picture to be coded. Alternatively, it may be described as headerinformation of the entire sequence, or as header information in units ofseveral pictures (e.g., in units of GOPs in MPEG).

Further, since the picture B11 is used as a reference picture whencoding a picture positioned backward the picture B11, informationindicating that decoded image data Dd corresponding to the picture B11should be stored in the reference picture memory 117 at decoding, isalso added as header information. Further, information indicating thatthe data Dd should be stored in the reference picture memory 117 untildecoding of the picture B12 is completed, is also added as headerinformation.

When all of the remaining blocks in the picture B11 have been coded,coding of the picture B12 takes place.

Coding Process for Picture B12

Since the picture B12 is a B picture, inter-picture predictive codingusing bidirectional reference is carried out as a coding process for thepicture B12. In this case, two I or P pictures which are timewise closeto the target picture B12, and a B picture which is timewise closest tothe target picture B12 are used as candidate pictures for forwardreference. Further, an I or P picture which is timewise closest to thetarget picture B12 is used as a candidate picture for backwardreference. To be specific, the pictures P7, P10, and B11 are used ascandidate pictures for forward reference for the picture B12, and thepicture P13 is used as a backward reference picture for the picture B12.

Since the picture B12 is not used as a reference picture when codinganother picture, the coding control unit 170 controls the respectiveswitches with the control signal Cs1 so that the switch 113 is turned ONand the switches 114 and 115 are turned OFF. Accordingly, the image dataMd corresponding to the block in the picture B12, which is read from theinput picture memory 101, is input to the motion vector detection unit108, the mode selection unit 179, and the difference calculation unit102.

The motion vector detection unit 108 detects a forward motion vector anda backward motion vector corresponding to the macroblock in the pictureB12, with reference to the pictures P7, P10, and B11 stored in thereference picture memory 117, as forward reference pictures, and thepicture P13 stored in the reference picture memory 117, as a backwardreference picture.

In this case, a most suitable reference picture is selected from amongthe pictures P7, P10, and B11, and detection of a forward motion vectoris carried out according to the selected picture. The detected motionvectors are output to the mode selection unit 179 and the bit streamgeneration unit 104. Further, information indicating which one of thepictures P7, P10, and B11 is referred to in detecting the forward motionvector (reference picture information) is also output to the modeselection unit 179.

The mode selection unit 179 determines a coding mode for the block inthe picture B12, using the motion vectors detected by the motion vectordetection unit 108. As a coding mode for the B picture, one of thefollowing coding modes is selected: intra-picture coding mode,inter-picture predictive coding mode using a forward motion vector,inter-picture predictive coding mode using a backward motion picture,and inter-picture predictive coding mode using bidirectional motionvectors.

The coding mode Ms determined by the mode selection unit 179 is outputto the bit stream generation unit 104. Further, prediction image dataPd, which is obtained from the reference picture according to the codingmode that is determined by the mode selection unit 179, is output to thedifference calculation unit 102 and the addition unit 106. However, whenthe intra-picture coding mode is selected, no prediction image data Pdis output.

Further, when the intra-picture coding mode is selected by the modeselection unit 179, the switches 111 and 112 are controlled in the samemanner as described for the coding process of the picture P13.

Hereinafter, a description will be given of a case where theinter-picture predictive coding mode is selected by the mode selectionunit 179.

In this case, the operations of the difference calculation unit 102, theprediction error coding unit 103, the bit stream generation unit 104,the prediction error decoding unit 105, and the coding control unit 170are identical to those described for the fifth embodiment.

When the coding mode is one performing forward reference, informationindicating which one of the pictures P7, P10, and B11 is referred to indetecting the forward motion vector (reference picture information) isalso added to the bit stream.

Further, information indicating that the target B picture B12 issubjected to inter-picture predictive coding using a forward B pictureB11 as a candidate for a reference picture is described as headerinformation. Furthermore, information indicating that the candidatepictures for forward reference are two I or P pictures and one B pictureis described as header information.

Moreover, information indicating that the picture B12 is not to be usedas a reference picture when coding the following pictures is added asheader information.

Thereby, it is easily determined that there is no necessity to store thedecoded image data Dd corresponding to the picture B12 in the referencepicture memory at decoding, whereby management of the reference picturememory is facilitated.

The above-mentioned header information may be described as headerinformation in units of pictures, i.e., as header information for everytarget picture to be coded. Alternatively, it may be described as headerinformation of the entire sequence, or as header information in units ofseveral pictures (e.g., in units of GOPs in MPEG).

The remaining blocks in the picture B12 are coded in the same manner asdescribed above.

Thereafter, the image data corresponding to the respective picturesfollowing the picture B12 are coded in like manner as described aboveaccording to the picture type. For example, P pictures are processedlike the picture P13, and the first B picture of the continuous Bpictures (picture B14, B17, or the like) is processed like the pictureP11. Further, the second B picture of the continuous B pictures (pictureB15, B18, or the like) is processed like the picture P12.

As described above, in the moving picture coding apparatus 70 accordingto the seventh embodiment, when coding a B picture as a target picture,since a B picture is also used as a candidate picture for forwardreference as well as P pictures, a forward reference picture that ispositioned closest to the target picture can be used as a forwardreference picture. Thereby, prediction accuracy in motion compensationfor a B picture can be increased, resulting in enhanced codingefficiency.

Moreover, when coding a B picture as a target picture, informationindicating whether or not the target picture is to be used as areference picture when coding (decoding) another picture is added asheader information. Further, when the target picture is used as areference picture when coding (decoding) another picture, informationindicating a period during which the target picture should be stored inthe reference picture memory is added. Therefore, when decoding the bitstream Bs outputted from the moving picture coding apparatus 70, thedecoding end can easily know which picture should be stored in thepicture memory and how long the storage period is, whereby management ofthe reference picture memory at decoding is facilitated.

In this seventh embodiment, when a target B picture is coded usinganother B picture as a reference picture, this is described as headerinformation of the target B picture. However, the header information isnot necessarily described in picture units. It may be described asheader information of the entire sequence, or as header information inunits of several pictures (e.g., GOP in MPEG).

In this seventh embodiment, motion compensation is performed in units ofmacroblocks each comprising 16 pixels (horizontal direction)×16 pixels(vertical direction), and coding of a prediction error image data isperformed in units of blocks each comprising 4 pixels (horizontaldirection)×4 (vertical direction), or in units of blocks each comprising8 pixels (horizontal direction)×8 (vertical direction). However, motioncompensation and coding of prediction error image data may be carriedout in units of image spaces, each comprising different number of pixelsfrom those mentioned above.

Further, in this seventh embodiment, a coding mode for a P picture isselected from among intra-picture coding mode, inter-picture predictivecoding mode using a motion vector, and inter-picture predictive codingmode using no motion vector, while a coding mode for a B picture isselected from among intra-picture coding mode, inter-picture predictivecoding mode using a forward motion vector, inter-picture predictivecoding mode using a backward motion vector, and inter-picture predictivecoding mode using bidirectional motion vectors. However, selection of acoding mode for a P picture or a B picture is not restricted to thatmentioned for the seventh embodiment.

Further, while this seventh embodiment employs an image sequence inwhich two B pictures are inserted between an I picture and a P pictureor between adjacent P pictures, the number of B pictures insertedbetween an I picture and a P picture or between adjacent P pictures inan image sequence may be other than two, for example, it may be three orfour.

Furthermore, while in this seventh embodiment three pictures are used ascandidate pictures for forward reference when coding a P picture, thenumber of forward reference candidate pictures for a P picture is notrestricted thereto.

Furthermore, while in this seventh embodiment two P pictures and one Bpicture are used as candidate pictures for forward reference when codinga B picture, forward reference candidate pictures to be used in coding aB picture are not restricted thereto. For example, forward referencecandidate pictures for a B picture may be one P picture and two Bpictures, or two P pictures and two B pictures, or three pictures whichare timewise closest to the target picture regardless of the picturetype.

When, in coding a B picture, only one picture that is closest to thetarget B picture is used as a reference picture, it is not necessary todescribe information indicating which picture is referred to in coding atarget block in the B picture (reference picture information), in thebit stream.

Further, in this seventh embodiment, when coding a B picture, a Bpicture which is positioned forward an I or P picture that is positionedforward and closest to the target B picture, is not referred to.However, when coding a B picture, a B picture which is positionedforward an I or P picture that is positioned forward and closest to thetarget B picture, may be used as a reference picture.

Embodiment 8

FIG. 35 is a block diagram for explaining a moving picture decodingapparatus 80 according to an eighth embodiment of the present invention.

The moving picture decoding apparatus 80 according to the eighthembodiment decodes the bit stream Bs outputted from the moving picturecoding apparatus 70 according to the seventh embodiment.

The moving picture decoding apparatus 80 is different from the movingpicture decoding apparatus 20 according to the second embodiment incandidate pictures for forward reference pictures to be referred to whencoding a P picture and a B picture, and decoding modes for a B picture.

That is, the moving picture decoding apparatus 80 is provided with,instead of the memory control unit 204 and the mode decoding unit 223according to the second embodiment, a memory control unit 284 and a modedecoding unit 283 which operate in different manners from thosedescribed for the second embodiment.

To be specific, the memory control unit 284 according to the eighthembodiment controls a reference picture memory 287 such that, whendecoding a P picture, three pictures (I or P pictures) which arepositioned forward the P picture are used as candidate pictures forforward reference, and when decoding a B picture, two pictures (I or Ppictures) which are positioned forward the B picture, a forward Bpicture that is closest to the B picture, and a backward I or P pictureare used as candidate pictures. However, a B picture which is positionedforward an I or P picture that is positioned forward and closest to thetarget picture, is not referred to.

The memory control unit 284 controls the reference picture memory 287,with a control signal Cm, on the basis of a flag indicating whether ornot the target picture is to be referred to in coding a picture thatfollows the target picture, which flag is inserted in the code strongcorresponding to the target picture.

To be specific, information (flag) indicating that the data of thetarget picture should be stored in the reference picture memory 287 atdecoding, and information indicating a period during which the data ofthe target picture should be stored, are included in the bit streamcorresponding to the target picture.

Further, when decoding a block (target block) in a P picture, the modedecoding unit 283 according to the eighth embodiment selects, as acoding mode for the target block, one from among the following modes:intra-picture decoding, inter-picture predictive decoding using a motionvector, and inter-picture predictive decoding using no motion vector (amotion is treated as zero). When decoding a block (target block) in a Bpicture, the mode decoding unit 283 selects, as a decoding mode for thetarget block, one from among the following modes: intra-picturedecoding, inter-picture predictive decoding using a forward motionvector, inter-picture predictive decoding using backward motion vector,and inter-picture predictive decoding using a forward motion vector anda backward motion vector. That is, the mode decoding unit 283 of themoving picture decoding apparatus 80 according to this eighth embodimentis different from the mode decoding unit 223 of the moving picturedecoding apparatus 20 according to the second embodiment only in that itdoes not use the direct mode, and therefore, the moving picture decodingapparatus 80 does not have the motion vector storage unit 226 of themoving picture decoding apparatus 20. Other constituents of the movingpicture decoding apparatus 80 according to the seventh embodiment areidentical to those of the moving picture decoding apparatus 20 accordingto the second embodiment.

Further, the moving picture decoding apparatus 80 according to theeighth embodiment different from the moving picture decoding apparatus60 according to the sixth embodiment in that the memory control unit 284controls the bit stream generation unit 104 so that a flag indicatingwhether or not the target picture is to be referred to in coding apicture after the target block is inserted in the bit streamcorresponding to the target picture. Further, in the moving picturedecoding apparatus 80, candidate pictures to be referred to in decodinga P picture and a B picture are also different from those employed inthe moving picture decoding apparatus according to the sixth embodiment.Other constituents of moving picture decoding apparatus 80 according tothe seventh embodiment are identical to those of the moving picturedecoding apparatus 60 according to the sixth embodiment.

Next, the operation of the moving picture decoding apparatus 80 will bedescribed.

The bit stream Bs outputted from the moving picture coding apparatus 70according to the seventh embodiment is input to the moving picturedecoding apparatus 80.

In this eighth embodiment, when decoding a P picture, three pictures (Ior P pictures) which are timewise forward and close to the P picture areused as candidates for a reference picture. On the other hand, whendecoding a B picture, two pictures (I or P pictures) which arepositioned timewise forward and close to the B picture, a B picturewhich is positioned forward and closest to the B picture, and an I or Ppicture which is positioned backward the target picture, are used ascandidate pictures for a reference picture. However, in decoding a Bpicture, a B picture which is positioned forward an I or P picture thatis positioned forward and closest to the target picture, is not referredto. Further, in decoding an I picture, other pictures are not referredto.

Further, information indicating which of the candidate pictures is usedas a reference picture in decoding a P picture or a B picture isdescribed as header information Ih of the bit stream Bs, and the headerinformation Ih is extracted by the bit stream analysis unit 201.

The header information Ih is output to the memory control unit 284. Theheader information may be described as header information of the entiresequence, header information in units of several pictures (e.g., GOP inMPEG), or header information in picture units.

The pictures in the bit stream Bs inputted to the moving picturedecoding apparatus 80 are arranged in order or picture decoding as shownin FIG. 36(a). Hereinafter, decoding processes for the pictures P13,B11, and B12 will be specifically described in this order.

Decoding Process for Picture P13

When the bit stream corresponding to the picture P13 is input to the bitstream analysis unit 201, the bit stream analysis unit 201 extractsvarious kinds of data from the inputted bit stream. The various kinds ofdata are information (coding mode) Ms relating to mode selection,information of the motion vector MV, the above-described headerinformation, and the like. The extracted coding mode Ms is output to themode decoding unit 283. Further, the extracted motion vector MV isoutput to the motion compensation decoding unit 205. Furthermore, thecoded data Ed extracted by the bit stream analysis unit 201 is output tothe prediction error decoding unit 202.

The mode decoding unit 283 controls the switches 209 and 210 withreference to the mode selection information (coding mode) Ms extractedfrom the bit stream. When the coding mode Ms is intra-picture codingmode and when the coding mode Ms is inter-picture predictive codingmode, the switches 209 and 210 are controlled in like manner asdescribed for the sixth embodiment.

Further, the mode decoding unit 283 outputs the coding mode Ms to themotion compensation decoding unit 205.

Hereinafter, a description will be given of the case where the codingmode is inter-picture predictive coding mode.

Since the operations of the prediction error decoding unit 202, themotion compensation decoding unit 205, and the addition unit 208 areidentical to those described for the sixth embodiment, repeateddescription is not necessary.

FIG. 37 shows how the pictures, whose data are stored in the referencepicture memory 207, change with time.

When decoding of the picture P13 is started, the pictures B8, P7, andP10 are stored in areas R1, R2, and R3 of the reference picture memory207. The picture P13 is decoded using the pictures P7 and P10 ascandidates for a reference picture, and the picture P13 is stored in thememory area R1 where the picture B8 had been stored. Such rewriting ofimage data of each picture in the reference picture memory is carriedout based on the header information of each picture which is added tothe bit stream. This header information indicates that the picture P7should be stored in the reference picture memory 207 until decoding ofthe picture P13 is completed, the picture P10 should be stored in thememory until decoding of the picture P16 is completed, and the pictureB8 should be stored in the memory until decoding of the picture B9 iscompleted.

In other words, since it can be decided that the picture B8 is notnecessary for decoding of the picture P13 and the following pictures,the picture P13 is written over the reference picture memory area R1where the picture B8 is stored.

Further, since information indicating that the picture P13 should bestored in the reference picture memory until decoding of the picture P19is completed is described as header information of the picture P13, thepicture P13 is stored in the reference picture memory at least untilthat time.

As described above, the blocks in the picture P13 are successivelydecoded. When all of the coded data corresponding to the blocks in thepicture P13 have been decoded, decoding of the picture B11 takes place.

Decoding Process for Picture B11

Since the operations of the bit stream analysis unit 201, the modedecoding unit 203, and the prediction error decoding unit 202 areidentical to those described for decoding of the picture P13, repeateddescription is not necessary.

The motion compensation decoding unit 205 generates motion compensationimage data Pd from the inputted information such as the motion vector.That is, the information inputted to the motion compensation decodingunit 205 is the motion vector MV and reference picture indexcorresponding to the picture B11. The picture B11 has been coded usingthe picture P10 as a forward reference picture, and the picture P13 as abackward reference picture. Accordingly, in decoding of the picture B11,these candidate pictures P10 and P13 have already been decoded, and thecorresponding decoded image data DId are stored in the reference picturememory 207.

When the coding mode is bidirectional predictive coding mode, the motioncompensation decoding unit 205 obtains a forward reference image fromthe reference picture memory 207 on the basis of the informationindicating the forward motion vector, and obtains a backward referenceimage from the memory 207 on the basis of the information indicating thebackward motion vector. Then, the motion compensation decoding unit 205performs addition and averaging of the forward reference image and thebackward reference image to generated a motion compensation image. DataPd of the motion compensation image so generated is output to theaddition unit 208.

The addition unit 208 adds the inputted prediction error image data PDdand motion compensation image data Pd to output addition image data Ad.The addition image data Ad so generated is outputted as decoded imagedata DId, through the switch 210 to the reference picture memory 207.

The memory control unit 284 controls the reference picture memory 207 onthe basis of information indicating which candidate pictures arereferred to in coding the P picture and the B picture, which informationis header information of the bit stream.

FIG. 37 shows how the pictures stored in the reference picture memory207 change with time.

When decoding of the picture P11 is started, the pictures P13, P7, andP10 are stored in the reference picture memory 207. The picture P11 isdecoded using the pictures P10 and P13 as reference pictures, and thepicture P11 is stored in the memory area R2 where the picture P7 hadbeen stored. Such rewriting of each picture in the reference picturememory 207 is carried out based on the header information of eachpicture which is added to the bit stream. This header informationindicates that the picture P7 should be stored in the reference picturememory 207 until decoding of the picture P13 is completed, the pictureP10 should be stored in the memory until decoding of the picture P16 iscompleted, and the picture P13 should be stored in the memory untildecoding of the picture P19 is completed.

In other words, since it is decided that the picture P7 is not necessaryfor decoding of the picture P13 and the following pictures, the pictureP11 is stored in the reference picture memory area R2 where the pictureP7 is stored.

Further, since information indicating that the picture B11 should bestored in the reference picture memory 207 until decoding of the pictureB12 is completed is described as header information of the picture B11,the picture B11 is stored in the reference picture memory 207 at leastuntil that time.

As described above, the coded data corresponding to the blocks in thepicture B11 are successively decoded. When all of the coded datacorresponding to the blocks in the picture B11 have been decoded,decoding of the picture B12 takes place.

Decoding Process for Picture B12

Since the operations of the bit stream analysis unit 201, the modedecoding unit 203, and the prediction error decoding unit 202 areidentical to those described for decoding of the picture P13, repeateddescription is not necessary.

The motion compensation decoding unit 205 generates motion compensationimage data Pd from the inputted information such as the motion vector.That is, the information inputted to the motion compensation decodingunit 205 is the motion vector MV and reference picture indexcorresponding to the picture B12. The picture B12 has been coded usingthe pictures P10 and B11 as candidates for a forward reference picture,and the picture P13 as a backward reference picture. These referencecandidate pictures P10, B11, and P13 have already been decoded, and thecorresponding decoded image data are stored in the reference picturememory 207.

When the coding mode is bidirectional predictive coding mode, the motioncompensation decoding unit 205 determined which one of the pictures P10and B11 is used as a forward reference picture in coding the pictureB12, according to the reference picture indices, and obtains a forwardreference image from the reference picture memory 207 according to theinformation indicating the forward motion vector. Further, the motioncompensation decoding unit 205 obtains a backward reference image fromthe memory 207 according to the information indicating the backwardmotion vector. Then, the motion compensation decoding unit 205 performsaddition and averaging of the forward reference image and the backwardreference image to generated a motion compensation image. Data Pd of themotion compensation image so generated is output to the addition unit208.

The addition unit 208 adds the inputted prediction error image data PDdand motion compensation image data Pd to output addition image data Ad.The addition image data Ad so generated is outputted as decoded imagedata DId, through the switch 210 to the reference picture memory 207.

The memory control unit 284 controls the reference picture memory 207 onthe basis of information indicating which reference pictures are used incoding the P picture and the B picture, which information is extractedfrom the header information of the bit stream.

FIG. 37 shows how the pictures stored in the reference picture memory207 change with time. When decoding of the picture B12 is started, thepictures P13, B11, and P10 are stored in the reference picture memory207. The picture B12 is decoded using the pictures P13, B11, and P10 asreference candidate pictures. Since information indicating that thepicture B12 is not to be used as a reference picture when decodinganother picture is described as header information, the decoded data ofthe picture B12 is not stored in the reference picture memory 207 butoutputted as output image data Od.

As described above, the coded data corresponding to the blocks in thepicture B12 are successively decoded. The decoded image data of therespective pictures which are stored in the reference picture memory207, and the decoded image data which are not stored in the referencepicture memory 207 are rearranged in order of their display times asshown in FIG. 36(b), and outputted as output image data Od.

Thereafter, the coded data corresponding to the respective pictures aredecoded in like manner as described above according to the picture type.

To be specific, the coded data of the P pictures are decoded like thepicture P13, and the first B picture (picture B14, B17, or the like) ofthe continuous B pictures is decoded like the picture P11. Further, thesecond B picture (picture B15, B18, or the like) of the continuous Bpictures is decoded like the picture P12.

As described above, in the moving picture decoding apparatus 80according to the eighth embodiment, since a B picture is used as areference candidate picture when decoding a B picture, a bit stream,which is obtained in a coding process that uses a B picture as well as Ppictures as forward reference candidate pictures when coding a Bpicture, can be accurately decoded. Further, since the reference picturememory is controlled using information obtained from the bit stream,indicating which reference pictures are used in coding a P picture and aB picture, the reference picture memory can be effectively utilized.That is, image data of pictures to be used as reference pictures in thefollowing decoding process are maintained in the reference picturememory, while image data of pictures not to be used as referencepictures in the following decoding process are successively erased fromthe memory, whereby the reference picture memory can be effectivelyutilized.

While this eighth embodiment employs a bit stream corresponding to animage sequence in which two B pictures are inserted between adjacent Ppictures, the number of B pictures positioned between adjacent Ppictures may be other than two, for example, it may be three or four.

Furthermore, while in this eighth embodiment two pictures are used ascandidate pictures for forward reference when decoding a P picture, thenumber of forward reference candidate pictures to be referred to indecoding a P picture is not restricted thereto.

Furthermore, in this eighth embodiment, when decoding a B picture, one Ppicture and one B picture are used as candidate pictures for forwardreference, and a B picture which is positioned forward an I or P picturethat is timewise closest to the target B picture, is not used as areference picture. However, pictures to be used as reference candidatepictures in decoding a B picture may be other than those described forthe eighth embodiment. Further, when decoding a B picture, a B picturewhich is positioned forward an I or P picture that is timewise closestto the target B picture, may be used as a reference picture.

Furthermore, while in the eighth embodiment decoded image data ofpictures which are not to be used as reference pictures when decodingother pictures are not stored in the reference picture memory, thedecoded image data of these pictures may be stored in the memory.

For example, when output of decoded image data of each picture iscarried out with a little delay from decoding of each picture, thedecoded image data of each picture must be stored in the referencepicture memory. In this case, a memory area, other than the memory areawhere the decoded image data of the reference candidate pictures arestored, is provided in the reference picture memory, and the decodedimage data of the pictures not to be used as reference pictures arestored in this memory area. Although, in this case, the storage capacityof the reference picture memory is increased, the method for managingthe reference picture memory is identical to that described for theeighth embodiment and, therefore, the reference picture memory can beeasily managed.

While all pictures are used as reference candidate pictures in thesecond, fourth, sixth, and eighth embodiments, all pictures are notnecessarily used as reference candidate pictures.

To be brief, in a moving picture decoding apparatus, usually,already-decoded pictures are once stored in a decoding buffer (decodedframe memory) regardless of whether they will be used as referencecandidate pictures or not, and thereafter, the already-decoded picturesare successively read from the decoding buffer to be displayed.

In the second, fourth, sixth, and eighth embodiments of the presentinvention, all pictures are used as reference candidate pictures and,therefore, all of already-decoded pictures are stored in a referencepicture memory for holding pictures to be used as reference candidatepictures, and thereafter, the already-decoded pictures are successivelyread from the reference picture memory to be displayed.

However, as described above, all of the already-decoded pictures are notnecessarily used as reference candidate pictures. Accordingly, thealready-decoded pictures may be once stored in a decoding buffer(decoded frame memory) for holding not only pictures not to be used asreference candidate pictures but also pictures to be used as referencecandidate pictures, and thereafter, the already-decoded pictures aresuccessively read from the decoding buffer to be displayed.

The moving picture coding apparatus or the moving picture decodingapparatus according to any of the aforementioned embodiments isimplemented by hardware, while these apparatuses may be implemented bysoftware. In this case, when a program for executing the coding ordecoding process according to any of the aforementioned embodiments isrecorded in a data storage medium such as a flexible disk, the movingpicture coding apparatus or the moving picture decoding apparatusaccording to any of the aforementioned embodiments can be easilyimplemented in an independent computer system.

FIGS. 38(a)-38(c) are diagrams for explaining a computer system forexecuting the moving picture coding process according to any of thefirst, third, fifth, and seventh embodiments and the moving picturedecoding process according to any of the second, fourth, sixth, andeighth embodiments.

FIG. 38(a) shows a front view of a flexible disk FD which is a mediumthat contains a program employed in the computer system, across-sectional view thereof, and a flexible disk body D. FIG. 38(b)shows an example of a physical format of the flexible disk body D.

The flexible disk FD is composed of the flexible disk body D and a caseFC that contains the flexible disk body D. On the surface of the diskbody D, a plurality of tracks Tr are formed concentrically from theouter circumference of the disk toward the inner circumference. Eachtrack is divided into 16 sectors Se in the angular direction. Therefore,in the flexible disk FD containing the above-mentioned program, data ofthe program for executing the moving picture coding process or themoving picture decoding process are recorded in the assigned storageareas (sectors) on the flexible disk body D.

FIG. 38(c) shows the structure for recording or reproducing the programin/from the flexible disk FD. When the program is recorded in theflexible disk FD, data of the program are written in the flexible diskFD from the computer system Csys through the flexible disk drive FDD.When the above-mentioned moving picture coding or decoding apparatus isconstructed in the computer system Csys by the program recorded in theflexible disk FD, the program is read from the flexible disk FD by theflexible disk drive FDD and then loaded to the computer system Csys.

Although in the above description a flexible disk is employed as astorage medium, an optical disk may be employed. Also in this case, themoving picture coding or decoding process can be performed by softwarein like manner as the case of using the flexible disk. The storagemedium is not restricted to these disks, and any medium may be employedas long as it can contain the program, for example, a CD-ROM, a memorycard, or a ROM cassette. Also when such data storage medium is employed,the moving picture coding or decoding process can be performed by thecomputer system in the same manner as the case of using the flexibledisk.

Applications of the moving picture coding method and the moving picturedecoding method according to any of the aforementioned embodiments andsystems using the same will be described hereinafter.

FIG. 39 is a block diagram illustrating an entire construction of acontents provision system 1100 that performs contents distributionservices.

A communication service provision area is divided into regions (cells)of desired size, and base stations 1107 to 1110 which are each fixedradio stations are established in the respective cells.

In this contents provision system 1100, various devices such as acomputer 1111, a PDA (personal digital assistant) 1112, a camera 1113, aportable telephone 1114, and a portable telephone with a camera 1200 areconnected to the Internet 1101 through an Internet service provider1102, a telephone network 1104, and the base stations 1107 to 1110.

However, the contents provision system 1100 is not restricted to asystem including all of the plural devices shown in FIG. 39, but may beone including some of the plural devices shown in FIG. 39. Further, therespective devices may be connected directly to the telephone network1104, not through the base stations 1107 to 1110 as the fixed radiostations.

The camera 1113 is a device that can take moving pictures of an object,like a digital video camera. The portable telephone may be a portabletelephone set according to any of PDC (Personal Digital Communications)system, CDMA (Code Division Multiple Access) system, W-CDMA(Wideband-Code Division Multiple Access) system, and GSM (Global Systemfor Mobile Communications) system, or PHS (Personal Handyphone System).

A streaming server 1103 is connected to the camera 1113 through the basestation 1109 and the telephone network 1104. In this system, livedistribution based on coded data which are transmitted by a user usingthe camera 1113 can be performed. The coding process for the data oftaken images may be carried out by either the camera 1113 or the serverthat transmits the data. Moving picture data which are obtained bytaking moving pictures of an object by means of the camera 1116 may betransmitted to the streaming server 1103 through the computer 1111. Thecamera 1116 is a device that can take still images or moving pictures ofan object, such as a digital camera. In this case, coding of the movingpicture data can be performed by either the camera 1116 or the computer1111. Further, the coding process is carried out by an LSI 1117 includedin the computer 1111 or the camera 1116.

Image coding or decoding software may be stored in a storage medium (aCD-ROM, a flexible disk, a hard disk, or the like) which is a recordingmedium that contains data readable by the computer 1111 or the like. Themoving picture data may be transmitted through the portable telephonewith a camera 1200. The moving picture data are data which have beencoded by an LSI included in the portable telephone 1200.

In this contents provision system 1100, contents corresponding to imagestaken by the user by means of the camera 1113 or the camera 1116 (forexample, live video of a music concert) are coded in the camera in thesame manner as any of the aforementioned embodiments, and transmittedfrom the camera to the streaming server 1103. The contents data aresubjected to streaming distribution from the streaming server 1103 to arequesting client.

The client may be any of the computer 1111, the PDA 1112, the camera1113, the portable telephone 1114 and the like, which can decode thecoded data.

In this contents provision system 1100, the coded data can be receivedand reproduced on the client side. When the data are received, decoded,and reproduced in real time on the client side, private broadcasting canbe realized.

The coding or decoding in the respective devices that constitute thissystem can be performed using the moving picture coding apparatus or themoving picture decoding apparatus according to any of the aforementionedembodiments.

A portable telephone will be now described as an example of the movingpicture coding or decoding apparatus.

FIG. 40 is a diagram illustrating a portable telephone 1200 that employsthe moving picture coding method and the moving picture decoding methodaccording to any of the aforementioned embodiments.

This portable telephone 1200 includes an antenna 1201 fortransmitting/receiving radio waves to/from the base station 1110, acamera unit 1203 that can take video or still images of an object, suchas a CCD camera, and a display unit 1202 such as a liquid crystaldisplay for displaying data of the video taken by the camera unit 1203or video received through the antenna 1201.

The portable telephone 1200 further includes a main body 1204 includingplural control keys, a voice output unit 1208 for outputting voices suchas a speaker, a voice input unit 1205 for inputting voices such as amicrophone, a recording medium 1207 for retaining coded data or decodeddata such as data of taken moving pictures or still images, or data,moving picture data or still image data of received e-mails, and a slotunit 1206 which enables the recording medium 1207 to be attached to theportable telephone 1200.

The recording medium 1207 has a flash memory element as a type of EEPROM(Electrically Erasable and Programmable Read Only Memory) that is anelectrically programmable and erasable non-volatile memory contained ina plastic case, like a SD card.

The portable telephone 1200 will be described more specifically withreference to FIG. 41.

The portable telephone 1200 has a main control unit 1241 that performsgeneral control for the respective units of the main body including thedisplay unit 1202 and the control key 1204.

The portable telephone 1200 further includes a power supply circuit1240, an operation input control unit 1234, an image coding unit 1242, acamera interface unit 1233, a LCD (Liquid Crystal Display) control unit1232, an image decoding unit 1239, a multiplexing/demultiplexing unit1238, a recording/reproduction unit 1237, a modulation/demodulation unit1236, and an audio processing unit 1235. The respective units of theportable telephone 1200 are connected to each other via asynchronization bus 1250.

The power supply circuit 1240 supplies power from a battery pack to therespective units when a call end/power supply key is turned ON under thecontrol of a user, thereby activating the digital portable telephonewith a camera 1200 to be turned into an operable state.

In the portable telephone 1200, the respective units operate undercontrol of the main control unit 1241 that is constituted by a CPU, aROM, a RAM and the like. To be more specific, in the portable telephone1200, an audio signal that is obtained by voice inputting into the voiceinput unit 1205 in a voice communication mode is converted into digitalaudio data by the audio processing unit 1235. The digital audio data issubjected to a spectrum spread process by the modulation/demodulationcircuit 1236, further subjected to a DA conversion process and afrequency transformation process by the transmission/receiving circuit1231, and transmitted through the antenna 1201.

In this portable telephone set 1200, a signal received through theantenna 1201 in the voice communication mode is amplified, and thensubjected to a frequency transformation process and an AD conversionprocess. The received signal is further subjected to a spectrum inversespread process in the modulation/demodulation circuit 1236, convertedinto an analog audio signal by the audio processing unit 1235, and thisanalog audio signal is outputted through the voice output unit 1208.

When the portable telephone 1200 transmits an electronic mail in a datacommunication mode, text data of the e-mail that is inputted bymanipulation of the control key 1204 on the main body is transmitted tothe main control unit 1241 via the operation input control unit 1234.The main control unit 1241 controls the respective units so that thetext data is subjected to the spectrum spread process in themodulation/demodulation circuit 1236, then subjected to the DAconversion process and the frequency transformation process in thetransmission/receiving circuit 1231, and then transmitted to the basestation 1110 through the antenna 1201.

When this portable telephone 1200 transmits image data in the datacommunication mode, data of an image taken by the camera unit 1203 issupplied to the image coding unit 1242 via the camera interface unit1233. When the portable telephone 1200 does not transmit the image data,the data of the image taken by the camera unit 1203 can be displayeddirectly on the display unit 1202 via the camera interface unit 1233 andthe LCD control unit 1232.

The image coding unit 1242 includes the moving picture coding apparatusaccording to any of the aforementioned embodiments. This image codingunit 1242 compressively encodes the image data supplied from the cameraunit 1203 by the moving picture coding method according to any of theabove embodiments to convert the same into coded image data, and outputsthe obtained coded image data to the multiplexing/demultiplexing unit1238. At the same time, the portable telephone 1200 transmits voiceswhich are inputted to the voice input unit 1205 while the image is beingtaken by the camera unit 1203, as digital audio data, to themultiplexing/demultiplexing unit 1238 through the audio processing unit1235.

The multiplexing/demultiplexing unit 1238 multiplexes the coded imagedata supplied from the image coding unit 1242 and the audio datasupplied from the audio processing unit 1235 by a predetermined method.Resultant multiplexed data is subjected to a spectrum spread process inthe modulation/demodulation circuit 1236, then further subjected to theDA conversion process and the frequency transformation process in thetransmission/receiving circuit 1231, and obtained data is transmittedthrough the antenna 1201.

When the portable telephone 1200 receives data of a moving picture filethat is linked to a home page or the like in the data communicationmode, a signal received from the base station 1110 through the antenna1201 is subjected to a spectrum inverse spread process by themodulation/demodulation circuit 1236, and resultant multiplexed data istransmitted to the multiplexing/demultiplexing unit 1238.

When the multiplexed data that is received via the antenna 1201 isdecoded, the multiplexing/demultiplexing unit 1238 demultiplexes themultiplexed data to divide the data into a coded bit streamcorresponding to the image data and a coded bit stream corresponding tothe audio data, and the coded image data is supplied to the imagedecoding unit 1239 and the audio data is supplied to the audioprocessing unit 1235, via the synchronization bus 1250.

The image decoding unit 1239 includes the moving picture decodingapparatus according to any of the aforementioned embodiments. The imagedecoding unit 1239 decodes the coded bit stream of the image data by thedecoding method corresponding to the coding method according to any ofthe above-mentioned embodiments, to reproduce moving picture data, andsupplies the reproduced data to the display unit 1202 through the LCDcontrol unit 1232. Thereby, for example, the moving picture dataincluded in the moving picture file that is linked to the home page isdisplayed. At the same time, the audio processing unit 1235 converts theaudio data into an analog audio signal, and then supplies the analogaudio signal to the voice output unit 1208. Thereby, for example, theaudio data included in the moving picture file that is linked to thehome page is reproduced.

Here, a system to which the moving picture coding method and the movingpicture decoding method according to any of the aforementionedembodiments is applicable is not restricted to the above-mentionedcontents provision system.

Recently, digital broadcasting using satellites or terrestrial waves istalked frequently, and the image coding apparatus and the image decodingapparatus according to the above embodiments is applicable also to adigital broadcasting system as shown in FIG. 42.

More specifically, a code bit stream corresponding to video informationis transmitted from a broadcast station 1409 to a satellite 1410 such asa communication satellite or a broadcast satellite, via radiocommunication. When the broadcast satellite 1410 receives the coded bitstream corresponding to the video information, the satellite 1410outputs broadcasting waves, and these waves are received by an antenna1406 at home including satellite broadcast receiving facility. Forexample, an apparatus such as a television (receiver) 1401 or a set topbox (STB) 1407 decodes the coded bit stream, and reproduces the videoinformation.

Further, the image decoding apparatus according to any of theaforementioned embodiments can be mounted also on a reproductionapparatus 1403 that can read and decode the coded bit stream recorded ona storage medium 1402 such as a CD or a DVD (recording medium).

In this case, a reproduced video signal is displayed on a monitor 1404.The moving picture decoding apparatus may be mounted on the set top box1407 that is connected to a cable for cable television 1405 or anantenna for satellite/terrestrial broadcast 1406, to reproduce an outputof the moving picture decoding apparatus to be displayed on a monitor1408 of the television. In this case, the moving picture decodingapparatus may be incorporated not in the set top box but in thetelevision. A vehicle 1412 having an antenna 1411 can receive a signalfrom the satellite 1410 or the base station 1107, and reproduce a movingpicture to display the same on a display device of a car navigationsystem 1413 or the like which is mounted on the vehicle 1412.

Further, it is also possible that an image signal can be coded by themoving picture coding apparatus according to any of the aforementionedembodiments and recorded in a recording medium.

A specific example of a recording device is a recorder 1420 such as aDVD recorder that records image signals on a DVD disk 1421, and a diskrecorder that records image signals on a hard disk. The image signalsmay be recorded on a SD card 1422. Further, when the recorder 1420includes the moving picture decoding apparatus according to any of theaforementioned embodiments, the image signals which are recorded on theDVD disk 1421 or the SD card 1422 can be reproduced by the recorder 1420and displayed on the monitor 1408.

Here, the structure of the car navigation system 1413 may include, forexample, the components of the portable telephone shown in FIG. 41 otherthan the camera unit 1203, the camera interface unit 1233 and the imagecoding unit 1242, and the same apply to the computer 1111, or thetelevision (receiver) 1401.

Further, as the terminal such as the portable telephone 1114, one ofthree types of terminals: a transmission-receiving type terminal havingboth of an encoder and a decoder, a transmission terminal having only anencoder, and a receiving terminal having only a decoder can be mounted.

As described above, the moving picture coding method or the movingpicture decoding method according to any of the aforementionedembodiments is applicable to any of the above-mentioned devices orsystems, whereby the effects as described in the above embodiments canbe obtained.

Moreover, it is needless to say that the embodiments of the presentinvention and the applications thereof are not restricted to thosedescribed in this specification.

APPLICABILITY IN INDUSTRY

As described above, in the moving picture coding method and the movingpicture decoding method according to the present invention, when atarget picture to be coded or decoded is a B picture, a forward picturethat is positioned closest to the target picture can be used as areference picture for the target picture, whereby prediction accuracy inmotion compensation for the B picture is increased, resulting inenhanced coding efficiency. Particularly, these methods are useful indata processing for transferring or recording moving picture data.

1. A moving picture decoding apparatus which decodes an input bit streamof a moving picture, wherein the input bit stream of the moving pictureis generated by: generating, on a picture-by-picture basis, either of(i) first information indicating that a target picture, which is one ofan I picture, a P picture, and B picture, can be a reference picture tobe referred to when coding at least one of P pictures following thetarget picture or coding at least one of B pictures following the targetpicture and (ii) second information indicating that the target picturecannot be a reference picture to be referred to when coding each of Ppictures following the target picture or coding each of B picturesfollowing the target picture; generating, on a picture-by-picture basis,(iii) third information indicating a plurality of candidate referencepictures, each of the candidate reference pictures being a candidate fora reference picture selected from among pictures for which only thefirst information is attached, when coding a target P picture or codinga target B picture; generating an assignment rule information of anadaptive assignment method indicating a rule to assign reference pictureindices to the candidate reference pictures between a default assignmentmethod of assigning reference picture indices to the candidate referencepictures according to an initialized rule and an adaptive assignmentmethod of assigning reference picture indices to the candidate referencepictures by changing reference picture indices assigned by the defaultassignment method; generating, on a block-by-block basis, (iv) fourthinformation indicating one specified reference picture to be referred towhen performing predictive coding on a target block included in thetarget P picture or indicating one or two specified reference picturesto be referred to when performing predictive coding on a target blockincluded in the target B picture; coding, on a block-by-block basis, thetarget block by using the one specified reference picture whenperforming predictive coding on the target block included in the targetP picture or by using the one or two specified reference pictures to bereferred to when performing predictive coding on the target blockincluded in the target B picture to generate coded block data; andadding the first information, the second information, the thirdinformation, an assignment rule information of the adaptive assignmentmethod and the fourth information to the coded block data, said movingpicture decoding apparatus comprising: a first and second informationextracting unit operable to extract, from the input bit stream on apicture-by-picture basis, either of (i) first information indicatingthat a target picture, which is one of an I picture, a P picture, and Bpicture, can be a reference picture to be referred to when decoding atleast one of P pictures following the target picture or decoding atleast one of B pictures following the target picture and (ii) secondinformation indicating that the target picture cannot be a referencepicture to be referred to when decoding each of P pictures following thetarget picture or decoding each of B pictures following the targetpicture; a third information extracting unit operable to extract fromthe input bit stream on a picture-by-picture basis, (iii) thirdinformation indicating a plurality of candidate reference pictures, eachof the candidate reference pictures being a candidate for a referencepicture selected from among pictures for which only the firstinformation is attached, when decoding a target P picture or decoding atarget B picture; a fourth information extracting unit operable toextract from the input bit stream on a block-by-block basis, (iv) fourthinformation indicating one specified reference picture to be referred towhen performing predictive decoding on a target block included in thetarget P picture or indicating one or two specified reference picturesto be referred to when performing predictive decoding on a target blockincluded in the target B picture; a storing unit operable to store, on apicture-by-picture basis, the target picture into a memory as acandidate reference picture only when the first information is extractedby the first and second information extracting unit; a reference pictureindex assigning unit operable to assign reference picture indices to thecandidate reference pictures between a default assignment method ofassigning reference picture indices to the candidate reference picturesaccording to an initialized rule and an adaptive assignment method ofassigning reference picture indices to the candidate reference picturesby changing reference picture indices assigned by the default assignmentmethod, based on an assignment rule information of the adaptiveassignment method extracted from the input bit stream; and a decodingunit operable to decode, on a block-by-block basis, the target block byusing the one specified reference picture corresponding to the fourthinformation when performing predictive decoding on the target blockincluded in the target P picture or by using the one or two specifiedreference pictures to be referred to corresponding to the fourthinformation when performing predictive decoding on the target blockincluded in the target B picture, wherein, when the target picture is aP picture and the target block included in the target P picture ispredictively decoded, one specified reference picture is specified on ablock-by-block basis, from among a plurality of candidate referencepictures which are stored in the memory, and the target block ispredictively decoded with reference to the one specified referencepicture, and when the target picture is a B picture and the target blockincluded in the target B picture is predictively decoded with referenceto one or two specified reference pictures, the one or two specifiedreference pictures are specified, on a block-by-block basis, from amonga plurality of candidate reference pictures which are stored in thememory, and the target block is predictively decoded with reference tothe one or two specified reference pictures.